supraj.dev

Interview questions & answers

AI system design

End-to-end designs for LLM platforms: requirements, architecture, failure modes, observability and cost.

  • 25 questions
  • 6 medium
  • 19 hard

25 questions

Question 1HardEnterprise RAG

Design an internal assistant that answers questions from company documents for 20,000 employees, where every answer must respect the document permissions people already have.

#

About 6 million documents across a wiki, a file share and a ticket system, roughly 200,000 changed per day. Peak 50 questions per second. Target: first token within 3 seconds at p95. Some spaces (HR, legal, M&A) are tightly restricted, and leadership asks what happens when someone loses access.

Show answer

Requirements to pin down first: which sources and how fresh (minutes or a day), the revocation target (for example, access removed in the source is enforced within 15 minutes), answer quality per department, and whether restricted spaces are in scope at launch.

Architecture: - Connectors per source pull content and ACLs together: the users and groups who can read each document. ACL-only changes are their own event, because a permission change does not change the text. - Ingestion parses, chunks and embeds. Each chunk stores the document ID, version and the list of principal IDs allowed to read it. - At query time the service resolves the user's identity and group memberships from the identity provider (cached for a few minutes), and passes them as a filter inside the vector and keyword queries. Filtering inside the query keeps recall; filtering the top 10 afterwards can leave nothing for a user with narrow access. - For restricted spaces, re-check the final cited documents against the source system before showing them. That closes the gap between an ACL change and the next index update. - The model sees only chunks the user may read, plus instructions to cite them. Answers and caches are keyed by user or by permission set, never shared across users.

Scale arithmetic: if documents average 10 chunks, that is 60 million vectors; at 1,024 float32 dimensions (4 KB each) about 246 GB of raw vectors before index overhead, so plan for sharding or quantization.

Failure modes and mitigations: - Group sync lag: someone removed from a group still retrieves its documents. Track sync lag as a metric with an alert. - Deleted documents still answer. Treat deletes as events with their own deadline, including caches and evaluation sets. - Injected text inside a shared document. Treat retrieved text as data, and keep tools out of this assistant at launch.

Observability: one trace per question with retrieved chunk IDs, the number of principals in the filter, stage latencies and the cited sources; metrics for connector freshness, ACL sync lag, empty-retrieval rate and citation coverage.

Security testing: plant canary documents restricted to a test group and, nightly, query as users outside it. Any hit is a release-blocking incident. OWASP lists permission-aware vector stores and strict partitioning as the first mitigation for vector and embedding weaknesses.

Rollout: one department with a labelled question set, then wider, adding restricted spaces last.

Trade-offs

Filtering inside the query with synced ACLs is fast but can lag behind the source; checking every result against the source is exact but adds latency and load, so reserve it for restricted spaces and final citations.

Common mistake

Indexing everything with a service account and asking the model not to reveal documents the user cannot see.

Follow-up

A user is added to a group and still cannot find a document for an hour. Where would you look first?

Question 2HardGateways

Design a central LLM gateway for 300 internal teams that call three model providers and one self-hosted model. It must enforce quotas, fall back during outages, and tell finance what each team spends.

#

Peak about 1,500 requests per second, most of them streamed. Teams currently hold their own provider keys. Finance wants monthly chargeback by team and feature, and security wants provider keys out of application code.

Show answer

Requirements: OpenAI-compatible and provider-native APIs, streaming, per-team quotas in tokens (not just requests), a policy for which data classes may go to which provider, failover rules, chargeback accuracy within a stated tolerance, and a latency overhead budget (for example, under 20 ms at p99).

Architecture: - Stateless gateway pods in several zones behind a load balancer. Teams authenticate with workload identity or gateway-issued keys; provider keys live only in the gateway's secret store. - Policy: per-team allowed models, data classification, maximum context and output length. - Admission: token buckets per team and per model in a shared store such as Redis. Reserve estimated input tokens plus max_tokens before the call, and reconcile with the reported usage when the response ends. Both OpenAI and Anthropic describe their limits in tokens and return remaining-capacity headers (x-ratelimit-remaining-tokens, anthropic-ratelimit-output-tokens-remaining and others) plus retry-after on 429. Read those to slow down before the provider starts rejecting. - Routing: a route table per model class (primary, secondary), with a circuit breaker per provider and region on error rate and time to first token. Fallback only to models that passed the same evaluation suite, and record which model actually answered. - Caching: pass provider prompt-caching controls through. An exact-match response cache is opt-in per route, scoped by team, and only for deterministic requests. - Metering: write one usage record per request (team, feature tag, model, input, cached input and output tokens, price-table version) to an append-only ledger. Compute cost from the ledger, and reconcile monthly with invoices.

Failure modes: - The quota store is down. Decide fail-open with conservative local per-pod limits, or fail-closed; write the decision down. Most companies fail open for a few minutes. - Retry storms during a provider brownout: the gateway owns retries with jitter and a retry budget, and clients should not retry on top. - A stream breaks mid-response: bill what the provider reported, and do not silently retry a partly delivered answer.

Observability: per route, request rate, errors by class (429, 5xx, timeout), time to first token, tokens per second, fallback rate, quota rejections by team, gateway overhead. Alert on fallback rate and on spend anomalies per team.

Rollout: start in metering-only mode (no enforcement) so teams see their numbers, then enforce quotas team by team, then remove direct provider keys.

Trade-offs

Central control gives one place for keys, quotas and cost data, but the gateway becomes a critical dependency and adds a hop; keep it stateless, multi-zone and thin.

Common mistake

Rate limiting by request count when providers limit, and bill, by tokens.

Follow-up

How would you attribute the cost of a shared, cached system prompt across the teams that use it?

Question 3HardAgent platforms

Design a platform that lets 40 internal teams run agents with tools (code execution, SaaS APIs, internal databases) without each team reinventing safety.

#

About 5,000 agent runs per hour. Some tools only read; others create tickets, send emails or change records. Security insists that a prompt-injected agent must not be able to exfiltrate data or take irreversible actions on its own.

Show answer

Requirements: a tool registry, per-tool authorization, sandboxed code execution, human approval for high-risk actions, full audit, budgets and a kill switch, and a developer experience that makes the safe path the easy one.

Architecture: - Tool registry: each tool declares its schema, owner, risk class (read, write, irreversible), the data it can touch, and its rate limits. - Agent runtime: runs the model loop with step, token and wall-clock budgets. It never holds long-lived credentials. - Policy check on every call: who the agent acts for, whether that user may do this, and argument constraints (for example, email only to internal domains). Deny by default. - Credential broker: issues short-lived, narrowly scoped tokens per call, injected by the executor, never placed in the prompt. - Code execution: a gVisor or Firecracker-class sandbox per run, with CPU, memory and time limits, an ephemeral filesystem and no network by default; any egress goes through an allowlisting proxy. - Approvals: irreversible actions pause the run and show a person the exact effect (recipient, amount, diff), with an expiry. Writes carry idempotency keys so retries do not duplicate them.

Threat model: an agent that combines private data, untrusted content and a way to send data out is exfiltration-capable (the lethal trifecta). The platform removes one leg per workflow; for example, runs that read untrusted web pages get no outbound email tool. OWASP calls the general failure excessive agency: too much functionality, permission or autonomy.

Observability: an execute_tool {gen_ai.tool.name} span per tool call (the OpenTelemetry GenAI convention) under the agent's trace; an audit log with the actor, tool, argument hash, policy decision, approver and result; metrics for denials, approvals pending, sandbox timeouts and cost per run. Alert on spikes in denials or egress blocks, which often mean injection attempts.

Rollout: read-only tools first, then writes behind approval, and loosen approval only per tool, with evidence from audit data.

Trade-offs

Central policy and brokered credentials add latency and integration work per tool, but they are the only controls that still hold when the model itself is manipulated.

Common mistake

Putting an API key in the system prompt and relying on instructions such as "only use this for read operations".

Follow-up

A team wants its agent to browse the web and also email customers. How would you allow that safely?

Question 4HardEvaluation platforms

Design an evaluation and regression pipeline that every prompt, model, or retrieval change must pass before it reaches production.

#

Six AI features, about 25 candidate changes a week. Each feature has 500 to 1,000 test cases. A full run with three samples per case and a model judge takes hours and costs real money, and engineers complain the gate is too slow.

Show answer

Requirements: a reproducible definition of a release candidate, per-feature metrics with agreed thresholds, a fast signal on pull requests, a decision rule that accounts for noise, and production checks after release.

Design: - Version everything as one candidate config: prompt, model ID and parameters, tools, retrieval settings, judge version. Hash it. - Datasets per feature: a curated golden set, a regression set built from past incidents, a privacy-filtered production sample refreshed monthly, and adversarial cases. Tag slices (language, document type, customer tier). - Graders: deterministic checks first (schema, citations resolve, forbidden content), then model judges calibrated against human labels, then a small human review sample. Re-check judge agreement whenever the judge prompt or model changes. - Statistics: compare candidate and baseline on the same items (paired), report confidence intervals, and gate on non-inferiority margins per metric and per critical slice, not on a single average. The error-bars paper argues for exactly this: report uncertainty and use paired tests on shared questions. - Tiers: a smoke tier on each pull request (about 100 cases, one sample, minutes), the full tier before release, and a nightly run on the current production config to catch provider drift. - Cost: cache outputs keyed by (config hash, case ID, sample index) and judge results keyed by (output hash, judge version); only changed components re-run.

Arithmetic: 6 features x 800 cases x 3 samples is 14,400 generations per full run, plus as many judge calls. At 25 candidates a week that is 360,000 generations if every change ran everything, which is why tiering and caching matter.

After release: shadow or canary traffic with online checks (refusal rate, schema failures, user feedback, sampled judge scores) and automatic rollback on guardrail breaches.

Observability: store every run with its config hash and per-slice results; dashboard trends; alert when the nightly production run drops or the judge disagrees with its human anchor set.

Trade-offs

A strict full gate catches more regressions but slows shipping and costs more; tiered gates with caching keep feedback fast while the full suite still runs before release.

Common mistake

Comparing a candidate's average score from one run with last month's number from a different dataset version.

Follow-up

How would you stop teams from overfitting prompts to the golden set over many iterations?

Question 5HardSelf-hosted inference

Design a self-hosted LLM inference platform on Kubernetes that serves four open models to twelve internal products and scales with demand.

#

Traffic swings from about 200 to 1,200 requests per second over the day. The cluster has H100 nodes; a new replica takes several minutes to load weights. Products have different latency targets, and one product runs nightly batch jobs.

Show answer

Requirements: latency SLOs per product (time to first token and inter-token latency), fairness between products, cost targets, and isolation of batch from interactive traffic.

Architecture: - GPU node pools per GPU type, tainted so only inference pods land there; node autoscaling with headroom for cold starts. - Weights in object storage with node-local caching or a streaming loader so new replicas start faster; keep a minimum number of warm replicas per model. - vLLM deployments per model (or per model and LoRA set), exposing Prometheus metrics. - Routing: the Kubernetes Gateway API Inference Extension adds model-aware routing through an InferencePool and an Endpoint Picker, which can use signals such as prefix-cache state and LoRA adapter availability instead of plain round robin. - An admission layer at the gateway with per-product quotas and priorities; batch work goes to a separate pool or lower priority so it cannot starve chat.

Autoscaling: GPU utilisation stays high whether healthy or overloaded, so scale on the waiting queue. vLLM exports vllm:num_requests_waiting, and KEDA's Prometheus scaler can turn it into a replica count: with the default AverageValue metric type, replicas are roughly the query result divided by the threshold. Add a slow scale-down so long streams are not cut off, and a terminationGracePeriodSeconds long enough to drain.

Failure modes: replicas that never become ready (bad weights, out of memory), node failures mid-stream, a hot model starving others, scale-up slower than the spike (keep headroom, or shed load early with clear 429s).

Observability: TTFT and inter-token latency histograms per model, queue length, KV cache usage (vllm:kv_cache_usage_perc), preemptions, replica startup time, GPU health counters; alert on SLO burn, not raw utilisation.

Rollout: one model and two products first, load-tested with recorded traffic, then add models.

Trade-offs

Warm minimum replicas cost GPU hours at night but protect latency; aggressive scale-to-zero saves money only where users accept a multi-minute cold start.

Common mistake

Scaling on GPU utilisation or CPU, which do not rise when users start queueing.

Follow-up

How would you pick the threshold of 4, and what would you load-test to confirm it?

KEDA ScaledObject scaling a vLLM Deployment on queue length (sketch) · yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: chat-model-scaler
spec:
  scaleTargetRef:
    name: chat-model-vllm          # the Deployment
  minReplicaCount: 2               # keep warm replicas; weights load slowly
  maxReplicaCount: 16
  cooldownPeriod: 300
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 600
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(vllm:num_requests_waiting{model_name="chat-model"})
        threshold: "4"             # about 4 waiting requests per replica
Question 6HardObservability

Design the observability stack for a company running fifteen LLM features: traces, token and cost metrics, quality signals, and alerts, without collecting sensitive prompts by default.

#

About 30 million model calls a month across hosted and self-hosted models. Each request can include retrieval, several model calls and tool calls. Security will not allow raw prompts in the general tracing backend.

Show answer

Requirements: debug a single bad answer end to end, know spend per feature, see latency and error SLOs, track quality over time, and keep sensitive content out of broadly accessible tools.

Instrumentation: OpenTelemetry with the GenAI semantic conventions, which are still marked Development (pin the version and expect renames). An inference span is named {gen_ai.operation.name} {gen_ai.request.model} with kind CLIENT; gen_ai.operation.name and gen_ai.provider.name are required, and usage attributes such as gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.usage.cache_read.input_tokens are recommended. Tool calls get execute_tool spans. Add your own attributes for feature, prompt version (gen_ai.prompt.name and gen_ai.prompt.version exist for named templates) and retrieved document IDs.

Content: message content attributes (gen_ai.input.messages, gen_ai.output.messages) are opt-in and the spec warns they are likely to contain personal data. Store content, when needed, in a restricted store with its own retention, and record only a reference: the spec defines _ref attributes holding an absolute URI to externally stored messages.

Pipeline: SDKs export to an OpenTelemetry Collector tier. Tail sampling keeps all error traces, slow traces and a percentage of the rest; it requires every span of a trace to reach the same Collector instance, so put a trace-ID-aware load-balancing layer in front.

Metrics: token counters (gen_ai.client.inference.usage.input_tokens, which includes cached tokens, and ...output_tokens) and duration histograms come from unsampled metrics. Cost = tokens x a versioned price table, computed in the metrics pipeline or warehouse.

Quality: user feedback, sampled judge scores and deterministic checks attached to traces (the conventions define a gen_ai.evaluation.result event).

Alerts: SLO burn on errors and time to first token per feature, spend anomalies per feature, spikes in length finish reasons, schema failures and refusals.

Volume arithmetic: 30 million calls a month is about 11.6 per second on average. At about 20 KB per trace that is 600 GB a month unsampled; keeping errors plus 10% is roughly a tenth of that.

Trade-offs

Capturing content makes debugging fast but creates a sensitive data store; references to a restricted store keep traces useful while limiting who sees text.

Common mistake

Building the cost dashboard from sampled traces, which undercounts spend by the sampling rate.

Follow-up

Which three alerts would you set up on day one, and which would you wait on until you have a baseline?

An inference span with GenAI attributes (OpenTelemetry Python API) · python
from opentelemetry import trace
from opentelemetry.trace import SpanKind, Status, StatusCode

tracer = trace.get_tracer("support-assistant")

def call_model(client, messages, feature):
    model = "chat-large-v3"
    with tracer.start_as_current_span(f"chat {model}", kind=SpanKind.CLIENT) as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", "openai")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("app.feature", feature)  # your own attribute
        try:
            resp = client.chat(model=model, messages=messages)  # your own client wrapper
        except Exception as ex:
            span.record_exception(ex)
            span.set_status(Status(StatusCode.ERROR))
            raise
        span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", resp.usage.output_tokens)
        return resp
Question 7HardIngestion

Design the document ingestion pipeline behind a search and RAG platform that holds 80 million documents and receives 1.5 million updates a day.

#

PDFs, HTML and office files from many sources. Search results must reflect a change within 30 minutes at p95, and deletions must disappear within an hour. Next quarter the team wants to switch to a new embedding model.

Show answer

Requirements: freshness and deletion targets, per-source throughput, supported formats, the ACL model, and a plan for reprocessing everything.

Pipeline: - Change capture: webhooks or change feeds where sources offer them, plus a periodic reconciliation crawl to catch missed events. Deletes and ACL changes are first-class events. - A durable queue partitioned by source, so one slow source does not block the others, with per-source rate limits. - Stages: fetch, parse (layout-aware for PDFs and tables), normalise, chunk, enrich (for example, a short contextual header per chunk), embed, write. Each stage is idempotent, keyed by (document ID, content hash, pipeline version). - Skip unchanged work: hash each chunk and re-embed only chunks whose hash changed. - Versioned writes: write the new chunk set, switch the document's current-version pointer, then delete the old chunks. Readers never see half a document. - A dead-letter queue for documents that fail repeatedly, with the parser, file type and error, so one corrupt file does not block a partition.

Arithmetic: 1.5 million updates a day is about 17 per second on average. With 12 chunks per document and 30% of chunks actually changed, that is 1.5M x 12 x 0.3 = 5.4 million embeddings a day, about 63 per second; size for peaks of several times that.

Re-embedding: 80 million documents x 12 chunks is 960 million chunks; at about 500 tokens each, roughly 480 billion tokens. Run it as a separate backfill into a new index (blue/green), dual-write fresh changes to both, evaluate retrieval on the new index, then switch reads. With a hosted model, batch endpoints suit this: OpenAI's Batch API accepts /v1/embeddings requests at a 50% discount with a 24-hour completion window.

Observability: freshness (source change to searchable) as a histogram per source, stage lag and throughput, failure rate by parser and file type, dead-letter depth, deletion lag. Alert on the freshness and deletion SLOs.

Security: parsers process untrusted files, so run them sandboxed with resource limits; carry ACLs with content; scan for malware.

Trade-offs

Content hashing and versioned writes add bookkeeping, but they make retries safe and cut embedding cost; a full rebuild is simpler to reason about but too slow for a 30-minute freshness target.

Common mistake

Updating chunks in place, so a failure midway leaves a document half old and half new in search results.

Follow-up

The reconciliation crawl finds 2% of documents missing from the index. How do you find out why?

Question 8HardReal-time voice

Design a real-time voice assistant for customer calls and build its latency budget: from the moment the caller stops talking to the first audio of the reply.

#

Target: under 800 ms at p50 and 1.5 s at p95. About 2,000 concurrent calls at peak. The assistant must sometimes look up an order with a tool call. A prototype feels sluggish even though the model's time to first token is fast.

Show answer

Requirements: the latency targets above, barge-in (the caller can interrupt), tool use, call recording and consent rules, and regional placement near the telephony provider.

Two architectures: - Speech-to-speech (a realtime model that takes and returns audio): fewer hops and lower latency; less control over each stage. - Cascaded (streaming speech-to-text, then an LLM, then streaming text-to-speech): more hops, but you can swap components, log text and run guardrails on text.

The budget, stage by stage (the numbers you choose are targets to measure, not facts): 1. End-of-turn detection. With silence-based detection you wait for a silence window before you even start. OpenAI's Realtime docs describe server_vad with a silence_duration_ms setting (their example uses 500) and note shorter values detect turns faster; semantic_vad instead decides from the words whether the user has finished. A 500 ms silence window alone uses most of an 800 ms budget, which is often why prototypes feel slow. 2. Final transcript (cascaded only): stream partial transcripts so the LLM can start on the final one immediately. 3. LLM time to first token: keep prompts short and cached, and start TTS on the first sentence, not the full reply. 4. TTS time to first audio, then network and jitter buffers.

Tool calls: say a short holding phrase while the lookup runs, and prefetch likely data (the customer's recent orders) when the call starts.

Barge-in: when the caller speaks, stop TTS playback, cancel generation, and record only what was actually played in the conversation history.

Capacity: 2,000 concurrent calls with a turn every 10 seconds or so means about 200 model requests per second at peak, each latency-sensitive.

Observability: per turn, timestamps for speech end, end-of-turn decision, final transcript, first token, first audio byte and playback start. Track each gap as its own percentile, and alert when any stage regresses.

Trade-offs

Speech-to-speech is fastest but harder to inspect and guard; cascaded pipelines give control and text logs at the cost of extra hops.

Common mistake

Optimising model time to first token while a fixed silence window dominates the user-perceived delay.

Follow-up

How would you tune end-of-turn detection for callers who pause mid-sentence without making everyone else wait?

Question 9MediumBatch inference

You must re-classify 30 million historical support tickets with a new taxonomy within a week using a hosted model. Design the backfill system.

#

Each ticket is about 800 input tokens with a 50-token JSON answer. The results feed a reporting table that analysts use daily. Nobody needs results in real time.

Show answer

Requirements: finish within 7 days, schema-valid output, a known cost, and no disruption to analysts while it runs.

Choose batch processing: both OpenAI and Anthropic charge 50% of standard prices for batch jobs. Anthropic's Message Batches API limits a batch to 100,000 requests or 256 MB, whichever comes first, says most batches finish within an hour, expires a batch whose processing does not complete within 24 hours (unfinished requests come back as expired), and keeps results for 29 days. OpenAI's Batch API has a 24-hour completion window and notes that output order may not match input order, so map results by custom_id.

Design: - A manifest table: ticket ID, input hash, prompt and taxonomy version, state (pending, submitted, succeeded, failed, expired), batch ID, attempt count. - A submitter packs pending rows into batches within the size limits and the provider's queued-request limits. - A poller downloads results soon after completion (not on day 28), validates each against the JSON schema, and writes rows idempotently keyed by (ticket ID, taxonomy version). - Failed and expired requests go back to pending with a capped retry count; persistent failures go to a review list. - Results land in a new table or column; the analyst view switches only when the backfill passes quality checks.

Arithmetic: 30 million tickets at 100,000 per batch is at least 300 batches, about 43 a day over 7 days. Tokens: 30M x 800 = 24 billion input and 30M x 50 = 1.5 billion output, so input cost dominates; trimming the prompt is worth more than shortening the answer.

Quality: before the full run, label 1,000 to 2,000 tickets by hand and measure accuracy per category. During the run, sample results per batch and stop if the error rate exceeds the agreed limit.

Observability: progress by state, success, error and expired counts per batch, schema-failure rate, spend so far against the estimate.

Trade-offs

Batch APIs halve cost and avoid rate-limit fights with live traffic, but give up latency guarantees and add orchestration; synchronous calls are simpler but cost double and compete with production.

Common mistake

Writing results in input order, or downloading them weeks later, after they have expired.

Follow-up

Halfway through, the taxonomy owner changes two category definitions. What do you rerun?

Question 10HardFine-tuning platforms

Design a fine-tuning platform where 15 teams can train LoRA adapters on open models using a shared pool of 64 GPUs, and ship them to production safely.

#

Base models from 8B to 70B parameters. Teams bring their own data, some of it containing customer text. Today teams run ad hoc scripts, GPUs sit idle at night, and nobody knows which dataset produced which adapter.

Show answer

Requirements: lineage from dataset to deployed adapter, fair GPU sharing with good utilisation, evaluation gates including safety, and cheap serving of many adapters.

Data: - A dataset registry with owner, source, licence, PII scan results and a content hash. Training jobs reference a registry version, never a path. - Deduplicate, split train and evaluation by time or by customer to avoid leakage, and block datasets that fail the PII policy.

Compute: - Kueue on Kubernetes: a ClusterQueue per team with a nominal GPU quota, grouped in a cohort so idle quota can be borrowed (with borrowing limits) and reclaimed through preemption. Jobs checkpoint regularly so preemption costs minutes, not days. - Job templates for supported base models and LoRA settings, so teams do not tune infrastructure.

Why LoRA: the LoRA paper reports that, compared with GPT-3 175B fine-tuned with Adam, it cut trainable parameters by 10,000 times and GPU memory by 3 times, with no additional inference latency (unlike adapter layers). Adapters are also small enough to keep many per base model.

Evaluation gate: the team's task suite, a shared regression suite, and a safety suite, compared against the base model. A fine-tune on harmless data can still weaken refusals, so safety checks are not optional.

Registry and serving: adapters stored with dataset version, base model, config and eval results, and signed. Serve with vLLM multi-LoRA (--enable-lora, with max_loras and max_lora_rank set for the fleet). vLLM's runtime adapter loading is behind VLLM_ALLOW_RUNTIME_LORA_UPDATING, and its docs warn it carries security risks and should not be used in production unless the environment is isolated and fully trusted; ship adapters through the normal release path instead.

Security: adapters can memorise training data, so they inherit the dataset's access controls; per-job service accounts and storage buckets per team.

Observability: queue wait per team, GPU utilisation per job, preemptions, failure reasons, cost per job (GPU-hours x rate), and eval results over adapter versions.

Trade-offs

Quota borrowing raises utilisation but means jobs can be preempted, so checkpointing becomes mandatory; hard per-team partitions are simpler but leave GPUs idle.

Common mistake

Letting teams load adapters into the shared production server at runtime without the evaluation and signing steps.

Follow-up

Two teams both claim the GPUs were idle when their job was preempted. What data settles it?

Question 11HardSearch

Design a hybrid semantic search service for an online store with 40 million products in 12 locales.

#

Peak 3,000 queries per second, p99 target 200 ms end to end. Prices and stock change every few minutes. Product managers want a cross-encoder reranker because it won an offline test.

Show answer

Requirements: the latency target, relevance measured on judged queries, filters (locale, stock, category), freshness for price and stock, and the zero-result rate.

Architecture: - Query understanding: normalisation, spelling, and extracting structured filters. - Two retrievers in parallel: BM25 for exact terms such as brand names and model numbers, and approximate nearest neighbour search on embeddings for meaning. Apply filters inside both, not after. - Fusion: reciprocal rank fusion combines rankings without needing comparable scores. - Reranking: a cross-encoder over the top candidates, within a fixed time budget. - Fast-changing fields (price, stock) live in a separate store or as filterable fields that update without re-embedding; embeddings change only when product text or images change. - Cache query embeddings and results for popular queries.

The reranker is the cost decision. At 3,000 QPS, reranking the top 100 means 300,000 query-document pairs scored per second, every second. Reranking the top 30 cuts that to 90,000; a smaller distilled reranker cuts the cost per pair. Measure quality at 20, 30, 50 and 100 candidates; the curve usually flattens well before 100.

A budget to measure against (targets, not facts): query parsing and embedding about 15 ms, parallel retrieval about 40 ms, fusion and feature fetch about 10 ms, reranking about 60 ms, leaving headroom for network and tail latency.

Evaluation: offline NDCG on judged queries per locale, then an online A/B test on conversion and revenue with guardrails on latency and zero-result rate.

Observability: latency per stage, timeouts that fall back to the fused list without reranking, zero-result rate per locale, click-through by position, and index freshness.

Trade-offs

Bigger rerank depth improves relevance with diminishing returns and linear cost; hybrid retrieval adds a second index but fixes exact-term misses that embeddings alone handle poorly.

Common mistake

Accepting an offline reranker win without checking its cost at peak QPS and its p99 latency.

Follow-up

If the reranker times out, what should the service return, and how would you know it is happening?

Question 12HardGuardrails

Design the content moderation and guardrail layer for a consumer chat product, covering both user input and model output.

#

3 million daily users, about 40 million messages a day in 9 languages, with images allowed. Product allows at most 150 ms of added latency at p95. Legal wants every block to be explainable, and support wants an appeals path.

Show answer

Requirements: the categories you enforce and their actions, latency budget, languages and modalities, explainability, appeals, and how you will measure false positives and false negatives.

Architecture: - Fast layer: rate limits, abuse signals, known-bad patterns. - Classifiers: a hosted moderation model or a self-hosted safety classifier. OpenAI's omni-moderation-latest accepts text and images (not audio) and its moderation endpoint is free to use; Llama Guard is an example of an LLM-based classifier that classifies both prompts and responses against a configurable risk taxonomy. - Policy engine: maps (category, score, context such as user age or surface) to an action (allow, warn, block, escalate). Policies and thresholds are versioned, so every decision can be explained as "policy v12, category X, score Y". - Output: streamed replies are moderated in chunks (for example, per sentence) before being shown, with the ability to stop and replace the rest of the reply. - Asynchronous review for borderline cases, a human review queue, and appeals that feed labelled data back.

Arithmetic: 40 million messages a day is about 463 per second on average. Checking input and output doubles that to about 926 classifications per second, and peaks of three times average mean about 2,800 per second. That sizes a self-hosted classifier fleet, or the rate limits you need from a hosted one.

Measurement: per language and category, precision from reviewer overturn rates, and recall from audits of sampled allowed traffic. A classifier tuned mostly on English often over-blocks or under-blocks other languages, so set thresholds per language where needed.

Observability: flag and block rates per category and language, added latency, classifier errors and timeouts (and whether you fail open or closed for each category), overturn rate. Alert on sudden shifts, which mean either an attack or a bad policy or model change.

Rollout: shadow mode first (decide but do not enforce), compare with human labels, then enforce category by category.

Trade-offs

Stricter thresholds reduce harmful output but raise false positives that hurt legitimate users; per-category, per-language thresholds cost more to maintain but avoid one global compromise.

Common mistake

Running moderation only on user input and trusting the model output.

Follow-up

The classifier times out for 2% of requests during a traffic spike. Which categories fail open and which fail closed?

Question 13HardResilience

Design multi-region serving for an AI assistant used in the US and the EU, where EU data must be processed in the EU, and the primary model provider has regional outages.

#

Availability target 99.9% per month for each region. The assistant uses a hosted model through a cloud provider plus a vector store. A recent outage in one region lasted 90 minutes and the team failed over to a global endpoint, which legal says was not allowed for EU traffic.

Show answer

Requirements: the availability target (99.9% of a 30-day month allows about 43 minutes of downtime, so a 90-minute outage blows the budget), residency rules per region, acceptable degraded modes, and recovery objectives for stateful data.

Know the endpoint types: - Azure's model deployment types say Global types may process data in any Azure region, Data Zone types process only within the Microsoft-specified zone (US, EU or Asia Pacific), and Standard and Regional Provisioned types process within the chosen geography. - Amazon Bedrock cross-region inference offers profiles tied to a geography such as the US or EU, where Bedrock picks a region within that geography, or global profiles where it picks any commercial region. A global endpoint is a good failover for the US only if US policy allows it; EU traffic may fail over only to EU-bound options.

Architecture: - Independent regional stacks: app, vector store and cache per region, with no cross-region calls on the request path. - Model access per region through an internal gateway with an ordered list of compliant targets: primary EU region, a second EU region or an EU data-zone endpoint, then a self-hosted or second-provider model in the EU. - Health: passive signals (error rate, time to first token) and synthetic probes drive a circuit breaker per target. - Capacity: the secondary must have quota for the full failover load. Pre-provision or reserve it; discovering limits during an outage is common. - Parity: every failover model passes the same evaluation suite; prompts are tested on it. - Degraded modes: a smaller model, cached answers to common questions, or disabling non-essential features.

Observability: requests by region and processing target, failover state, error budget burn per region, and an audit log recording where each request was processed, so residency can be proven later.

Rollout and drills: test failover regularly in game days, with legal sign-off on the target list.

Trade-offs

Residency-constrained failover has fewer and more expensive targets than a global endpoint; reserving secondary capacity costs money even when unused, but buys the availability target.

Common mistake

Adding a global endpoint as the universal fallback without checking where it processes data.

Follow-up

How would you prove to an auditor which region processed a given EU request last month?

Question 14HardPrivacy

Design the privacy architecture for an LLM assistant that handles health and identity data: PII redaction, retention, deletion requests, and observability without leaking content.

#

Users type free text that includes names, dates of birth and diagnoses. The assistant uses a hosted model, RAG over customer records, logs and traces, and an evaluation dataset built from production samples. Compliance asks you to handle a deletion request within 30 days across every system.

Show answer

Requirements: which data categories are allowed to reach which systems, retention per store, the deletion SLA, and provider contracts.

Data map: the request path, provider-side logs, traces, the vector index, caches, analytics, evaluation sets, and any fine-tuning data. Every copy needs an owner, a retention period and a deletion path.

Redaction and pseudonymisation: - Detect PII with a tool such as Microsoft Presidio (analyzers plus custom recognisers for your identifiers). Its README is explicit that automated detection cannot guarantee finding all sensitive data, so redaction is one layer, not a guarantee. - On the request path, replace identifiers with stable placeholders and keep the mapping in a restricted vault, so the model reasons over "PATIENT_1" and the response is re-identified only for the authorised user. - For logs and analytics, use irreversible redaction or drop content entirely.

Provider side: by default OpenAI generates abuse-monitoring logs that may contain prompts and responses; Zero Data Retention and Modified Abuse Monitoring require approval, and stateful endpoints differ in what they store. Read the provider's data-controls page per endpoint before choosing features.

Observability: traces carry metadata, not text. Where content is needed for debugging, store it in a restricted store with short retention and reference it from the span; the OpenTelemetry GenAI conventions make message content opt-in, warn it is likely to contain personal data, and define _ref attributes for externally stored messages.

Deletion: one deletion workflow keyed by user ID fans out to every store, including vector chunks, caches and evaluation sets, and records completion per store. Avoid fine-tuning on raw personal data: removing it from a trained model is not practical.

Testing: seed synthetic PII canaries in test traffic, measure redaction recall by entity type and language, and check that canaries never appear in logs or traces.

Trade-offs

Aggressive redaction protects data but can hurt answer quality when names or dates matter; reversible placeholders preserve meaning at the cost of running a sensitive mapping store.

Common mistake

Redacting the prompt while traces, caches and the evaluation set still keep raw copies.

Follow-up

A deletion request arrives for a user whose conversation was sampled into last month's evaluation set. What happens to that eval set and its historical scores?

Question 15HardEmbeddings

Design how a recommendation and search system refreshes its embeddings, including moving all items and users to a new embedding model without downtime.

#

200 million items and 50 million users with embeddings, served from a feature store and a vector index. Item text changes daily; user embeddings are recomputed nightly. The new model has better offline results.

Show answer

Requirements: freshness per entity type, the migration window, rollback, and consistency between training and serving.

Principles: - An embedding is only comparable with embeddings from the same model and version, so the version is part of every key, index name and feature definition. Query-side embeddings must come from the same version as the index they search. - Never mix versions in one index.

Feature store: a store such as Feast separates an offline store (historical features for training, joined point-in-time correct) from an online store (low-latency serving), and moves batch features online through materialisation. Train ranking models on the same embedding version you will serve, or you get training/serving skew.

Migration: 1. Backfill new-version item embeddings into a new index while the old one serves. 2. Dual-write: new or changed items are embedded with both models until cutover. 3. Recompute user embeddings with the new model; downstream rankers that use embeddings as features are retrained on them. 4. Evaluate offline, then A/B test on a traffic slice; route each request entirely to one version (query encoder, index and ranker together). 5. Ramp up, keep the old index warm for rollback, then retire it.

Arithmetic: 200 million items x 768 dimensions x 4 bytes is about 614 GB of raw float32 vectors; running two versions during migration doubles that to about 1.2 TB, before index overhead. Int8 or binary quantization with rescoring can cut that substantially.

Observability: a counter for version mismatches between query and index (should be zero), coverage (share of items with a current embedding), freshness lag per entity type, and stability checks such as overlap of top-k neighbours for a fixed probe set across refreshes.

Trade-offs

Dual-running doubles storage and embedding cost for the migration window, but gives instant rollback; an in-place swap is cheaper and leaves no safe way back.

Common mistake

Re-embedding items with the new model while queries still use the old model, which silently breaks retrieval.

Follow-up

Halfway through the backfill, item text keeps changing. How do you guarantee no item ends up with a stale new-version embedding?

Question 16MediumSLOs

Define SLOs and alerting for an internal LLM serving API used by 40 products, so on-call gets paged for real problems and not for noise.

#

The API fronts several self-hosted models. Today there is a static alert on "error rate above 1% for 5 minutes", which fires often at night with low traffic and missed a slow two-day degradation in time to first token.

Show answer

SLIs: define a good request from the user's point of view: it returned a valid response (not 5xx, not a timeout, and, for structured endpoints, valid against the schema) and its time to first token was under the threshold for that product class. Retries that the gateway hid from the caller still count as good only if the user-visible latency stayed within target.

SLOs: for example 99.9% of requests good over 30 days, possibly per product tier.

Alerting: the Google SRE workbook recommends multiwindow, multi-burn-rate alerts. For a 99.9% SLO its starting parameters are: page when 2% of the budget burns in 1 hour (burn rate 14.4, with a 5-minute short window), page when 5% burns in 6 hours (burn rate 6, 30-minute short window), and open a ticket when 10% burns in 3 days (burn rate 1, 6-hour short window). The short window makes the alert stop soon after the problem ends.

Why this fixes both problems: low-traffic noise at night does not burn 2% of a month's budget in an hour, and a slow degradation shows up in the 3-day ticket rule.

Latency SLI from vLLM: use the vllm:time_to_first_token_seconds histogram, and pick a threshold that is an actual bucket boundary so the ratio is exact.

Low-traffic products: burn-rate alerts are unreliable with very few requests; use synthetic probes or aggregate small products into one SLO.

Also alert on leading indicators without paging: queue length, KV cache pressure, and preemptions.

Trade-offs

Burn-rate alerts need more setup and good SLIs, but they page in proportion to user impact; static thresholds are simple but either too noisy or too slow.

Common mistake

Counting only HTTP 5xx as bad while users suffer from slow first tokens and truncated JSON.

Follow-up

How would you set separate SLOs for interactive and batch callers on the same API?

Prometheus rules: fast-burn page for a 99.9% SLO (sketch) · yaml
groups:
  - name: llm-api-slo
    rules:
      - record: slo:bad_ratio:rate1h
        expr: |
          sum(rate(llm_requests_total{good="false"}[1h]))
          / sum(rate(llm_requests_total[1h]))
      - record: slo:bad_ratio:rate5m
        expr: |
          sum(rate(llm_requests_total{good="false"}[5m]))
          / sum(rate(llm_requests_total[5m]))
      - alert: LLMApiErrorBudgetFastBurn
        expr: slo:bad_ratio:rate1h > (14.4 * 0.001) and slo:bad_ratio:rate5m > (14.4 * 0.001)
        labels:
          severity: page
Question 17HardAnalytics assistants

Design a natural-language analytics assistant that lets 3,000 business users query a BigQuery warehouse with 4,000 tables.

#

Users ask questions such as "revenue by region last quarter". Some tables contain salary and customer data with row-level restrictions. Finance worries about runaway query costs, and analysts worry about confidently wrong numbers.

Show answer

Requirements: accuracy on common business questions, permissions identical to what the user already has, bounded cost per query, and answers that show how they were computed.

Architecture: - Semantic layer: curated views and metric definitions (what "revenue" means, which date column, which filters), with descriptions. Retrieval selects the relevant views and examples for each question instead of putting 4,000 schemas in the prompt. - Generation: the model returns structured output (SQL plus a short explanation and assumptions) against a JSON schema; the application still validates it. - Validation before execution: parse the SQL, allow only SELECT on allowed views, add a LIMIT, then dry-run it to estimate bytes. - Cost cap: set BigQuery's maximum bytes billed per query; if the estimate is above the limit the query fails without incurring a charge. - Execute with the user's own identity, so the warehouse enforces row-level and column-level security; never with a shared superuser account. - Answer with the result, the SQL and the assumptions, so analysts can check them.

Accuracy: a golden set of questions with expected results, scored by comparing result sets (execution accuracy), not SQL text, because many different queries are correct. Add questions that must be refused or clarified ("revenue" without a period).

Observability: per question, retrieved views, generated SQL, validation outcome, bytes estimated and billed, latency, and whether the user corrected or re-asked. Alert on rising validation failures and on cost-cap hits.

Rollout: one domain (sales) with its own semantic layer first, then expand domain by domain.

Trade-offs

A curated semantic layer limits which questions can be answered but makes answers far more reliable; open access to raw tables answers more but is wrong more often and harder to secure.

Common mistake

Running generated SQL with a service account that can read every table, and relying on the prompt to respect permissions.

Follow-up

A user asks a question that the semantic layer cannot answer. What should the assistant do?

Question 18MediumDocument AI

Design a pipeline that extracts 30 fields from 1.2 million supplier invoices a month in many layouts, with humans reviewing only what needs review.

#

Critical fields (amount, currency, IBAN, due date) must be 99% correct after review; other fields 95%. Today a team keys every invoice by hand. Finance wants to know how many reviewers they will still need.

Show answer

Requirements: per-field accuracy targets, turnaround time, audit trail, and reviewer capacity.

Pipeline: 1. Ingest and classify (invoice, credit note, other) and detect language. 2. Extract with OCR plus an LLM, or a vision-language model, returning JSON against a schema. Structured outputs constrain completed, non-refusal output to the schema, but the application still has to handle refusals, incomplete responses and errors. 3. Validate with rules: line items sum to the total, currency matches the supplier, IBAN checksum, due date after invoice date, supplier exists in the vendor master. 4. Route: auto-accept when all critical fields pass validation and confidence is high; otherwise send to a review queue that shows the document region next to each field. 5. Store corrections with the original output; they become evaluation and training data.

Reviewer arithmetic: 1.2 million invoices a month is about 40,000 a day. If 8% go to review at 1.5 minutes each, that is 3,200 invoices and 4,800 minutes, or 80 reviewer-hours a day: about 10 people working 8-hour days. Each point of auto-accept rate gained removes about 10 reviewer-hours a day, which tells you where to invest.

Evaluation: per-field precision and recall on a labelled set stratified by supplier and layout, with critical fields reported separately; also measure how often auto-accepted invoices are later corrected, which is the real error rate.

Observability: auto-accept rate, review queue age, validation failures by rule, per-field correction rate, and drift by supplier (a new layout shows up as a spike in corrections).

Trade-offs

A higher auto-accept threshold lowers errors but raises reviewer cost; per-field thresholds let you be strict on money fields and lenient on descriptions.

Common mistake

Measuring only whole-document exact match, which hides which fields fail and over-routes to review.

Follow-up

Corrections show one large supplier changed its layout. How do you detect that within a day, not a month?

Question 19MediumCapacity and cost

Design a chat assistant product for 1 million daily active users and estimate its model traffic and token cost before you pick a model.

#

Each user sends about 20 messages a day. Each request carries a 2,000-token system prompt and tool definitions plus about 1,000 tokens of conversation, and replies average 300 tokens. Peak traffic is three times the daily average. The provider supports prompt caching.

Show answer

Traffic: 1 million users x 20 messages = 20 million requests a day, about 231 per second on average (20,000,000 / 86,400) and about 700 per second at peak.

Tokens per day: input 20M x 3,000 = 60 billion tokens; output 20M x 300 = 6 billion tokens. Even though output tokens usually cost more per token, input volume is ten times larger, so input dominates unless the output price is more than ten times the input price.

Caching: the 2,000-token system prompt and tools are the same for every request, so put them first and cache them. Anthropic, for example, prices cache reads at 0.1 times the base input price and 5-minute cache writes at 1.25 times. If 2,000 of 3,000 input tokens are read from cache, the input cost per request is about 1,000 + 2,000 x 0.1 = 1,200 token-equivalents instead of 3,000, a 60% cut on input (ignoring the occasional write). Caching the earlier conversation turns too saves more for long chats.

Architecture: - Clients stream over server-sent events or WebSockets through an API layer with per-user rate limits and abuse controls. - A conversation service stores messages and builds the context: system prompt, recent turns, a running summary of older turns, retrieved memories. - A model gateway handles provider routing, retries, quotas and metering. - Moderation on input and output; feature flags for model and prompt versions.

Failure modes: provider 429s at peak (negotiate limits for 700 RPS plus headroom, and shed low-priority traffic first), long conversations that blow the context window (summarise), cache misses after prompt edits (version prompts and expect a temporary cost rise).

Observability: requests, tokens and cache-read share per model and feature; cost per daily active user; time to first token; error and refusal rates; user feedback. Alert on a drop in cache-read share, which raises cost quietly.

Trade-offs

Caching the static prefix is nearly free to adopt and cuts input cost a lot; summarising old turns saves more tokens but can lose details users refer back to.

Common mistake

Estimating cost from output tokens only, or putting per-user data at the start of the prompt so nothing can be cached.

Follow-up

Product wants to put the user's name and preferences in the system prompt. What does that do to caching, and where would you put them instead?

Question 20HardAgents in production

Design a customer support agent that resolves conversations end to end, including actions such as refunds and address changes, with handoff to humans.

#

2 million conversations a month. Refunds under a set amount may be automatic; larger ones need an agent's approval. The business measures containment (resolved without a human) but also cares about wrong refunds and customer satisfaction.

Show answer

Requirements: which intents are in scope, which actions are allowed automatically and under what limits, handoff rules, and the metrics: resolution rate, wrong-action rate, customer satisfaction, cost per resolved conversation.

Architecture: - Intent routing: classify the request; out-of-scope or high-risk intents go straight to humans. - Knowledge: RAG over policies and help articles, with citations. - Actions through tools that enforce business rules in code: the refund tool checks order ownership, refund history and the amount limit itself, whatever the model asks for. Above the limit it creates an approval request with the exact amount and order, and the conversation tells the customer what happens next. - Identity: the customer is authenticated by the channel; tools act only on that customer's records. - Handoff: a human receives a summary, the conversation, actions taken and what is pending, so the customer does not repeat themselves. - Durable state: conversation and pending approvals survive restarts, and write actions use idempotency keys.

Evaluation before launch: simulated conversations with a model playing the customer, graded on final database state against policy. tau-bench is a published example of this setup (simulated users, domain API tools and policy rules) and introduced pass^k to measure whether an agent succeeds consistently across repeated trials, which matters more than a single lucky success when it handles money.

Metrics in production: containment and resolution, wrong-action rate from audits of sampled actions, repeat contacts within 7 days (a sign of false resolution), satisfaction, handoff reasons.

Observability: a trace per conversation with tool calls, policy decisions and approvals; alerts on refund volume or value anomalies and on handoff spikes.

Rollout: one intent (order status, read-only) first, then address changes, then small refunds, each with a kill switch.

Trade-offs

Higher automation limits raise containment and cut cost, but raise the cost of each mistake; start low and raise limits with audit evidence.

Common mistake

Optimising containment alone, which rewards an agent for ending conversations without solving the problem.

Follow-up

Customers learn to phrase requests so the agent always issues the maximum automatic refund. How would you detect and stop that?

Question 21HardAgent infrastructure

Design the job system behind a research agent whose tasks run 5 to 60 minutes and make dozens of model and tool calls each.

#

About 10,000 jobs a day, averaging 20 minutes. Users close the browser and come back later. Deploys happen several times a day, and today a deploy kills every running job.

Show answer

Requirements: jobs survive deploys and crashes, users can watch progress and cancel, per-user and per-tenant concurrency limits, cost budgets per job, and results stored for later.

Capacity: 10,000 jobs x 20 minutes = 200,000 job-minutes a day, which over 1,440 minutes is about 139 jobs running at any moment on average; plan for peaks several times higher. Most of that time is waiting on models and tools, so workers are I/O-bound and cheap, but model rate limits are shared and must be managed centrally.

Architecture: - Submission API writes a job record and starts a workflow; it returns a job ID immediately. - Durable execution: a workflow engine such as Temporal records an event history and recreates the pre-failure state after a crash, so a job resumes where it stopped. Model and tool calls run as activities with timeouts and retry policies, keeping side effects outside the replayed workflow code. - Budgets: maximum steps, tokens, wall-clock time and spend per job, enforced by the workflow. - Concurrency: per-tenant limits on running jobs, plus a shared rate limiter in front of model providers so a burst of jobs does not cause 429 storms. - Progress: the workflow emits events (step started, source found, draft ready) to a store that the UI reads or streams. - Cancellation: a signal that stops new steps and cancels running activities. - Results: stored with sources and the job trace ID.

Deploys: workers drain and in-flight workflows continue on new workers; changing workflow logic needs the engine's versioning approach, because old histories replay against new code.

Observability: job counts by state, queue wait time, duration and cost per job, step failure reasons, stuck jobs (no progress for N minutes) as an alert, and a trace per job (link spans per step rather than one giant span).

Trade-offs

A workflow engine adds infrastructure and a programming model to learn, but it removes a whole class of lost-work bugs; a plain queue with ad hoc checkpoints is simpler until the first deploy loses jobs.

Common mistake

Running each job inside the web request or a single worker process with state only in memory.

Follow-up

A job has been stuck for 30 minutes on a tool that never returns. What should the system do on its own?

Question 22MediumRelease engineering

Design how prompts and model configurations are versioned, released and rolled back across a dozen AI features.

#

Prompts are strings in application code today. A bad prompt edit last month needed a full deploy to revert, which took 40 minutes. Product managers want to edit prompts without engineers, and security wants an audit trail.

Show answer

Requirements: rollback in seconds without a deploy, an evaluation gate before any change goes live, gradual rollout, audit of who changed what, and the ability to tie every trace to the exact prompt version.

Design: - A prompt registry: each prompt is a template plus model ID, parameters, tool set and output schema, stored as an immutable version with an author, reason and evaluation results. - Applications reference prompts by name and resolve the active version at runtime from a configuration service with local caching (and a safe default if the service is unreachable). - Publishing a version runs the feature's evaluation suite automatically; failing versions cannot be activated. - Rollout: activate a version for a percentage of traffic or for internal users first, with guardrail metrics; rollback means pointing at the previous version. - Permissions: product managers can draft and test; activation in production needs a reviewer, like a code review. - Templates take variables as data; user input is never concatenated into the instruction part.

Observability: put the prompt name and version on every model span. The OpenTelemetry GenAI conventions define gen_ai.prompt.name (conditionally required when a named prompt template is used) and gen_ai.prompt.version, so dashboards can compare quality, cost and latency by version and spot which version a bad answer came from.

Failure modes: the configuration service is down (serve the cached version), a version references a model that was retired (validate at publish time), and two features share a prompt and one change breaks the other (separate prompts or run both suites).

Trade-offs

A registry adds a runtime dependency and process, but turns a 40-minute revert into a seconds-long switch; prompts in code are simpler and get code review for free, but couple every edit to a deploy.

Common mistake

Letting anyone hot-edit the live prompt with no evaluation run and no version history.

Follow-up

How would you run an A/B test between two prompt versions and decide which one wins?

Question 23HardRegulated deployments

A bank wants an LLM assistant that runs entirely inside an air-gapped network. Design how models get in, how it runs, and how it is operated with no internet access.

#

No outbound connectivity at all. Models, containers and updates must pass through a controlled transfer process with security review. About 5,000 users; the bank has its own GPU servers.

Show answer

Requirements: which models are approved, the transfer and review process, update cadence, capacity for 5,000 users, and local monitoring with no vendor telemetry.

Supply chain: - Approve specific model versions with their licence reviewed. Bring in weights with recorded checksums or signatures verified on both sides of the transfer. - Prefer weight formats that cannot run code when loaded: the safetensors README describes the format as safe to load, in contrast to pickle, which can run arbitrary code. Refuse pickle-based checkpoints, or convert them in a sandbox outside the network first. - Mirror container images, Python packages and drivers into internal registries, with vulnerability scans before import.

Runtime: - An internal inference service (for example vLLM) on the bank's GPUs, behind an internal gateway with authentication, quotas and audit logging. - Retrieval over internal documents with permission filtering, as in any enterprise assistant. - Disable every phone-home feature: usage statistics, update checks, remote model downloads. Configure servers to load models only from local paths, and block egress at the network layer as well.

Operations: - Local observability: Prometheus metrics, logs and traces stored inside the network. - An evaluation suite that runs inside the network, so each imported model version is tested before activation. - An update calendar: a monthly import window for patches and models, plus an emergency path for critical security fixes.

Capacity: size from expected concurrent users and tokens per request, then load-test on the actual hardware, since there is no cloud burst capacity to fall back on.

Trade-offs

An air gap removes whole classes of data-exfiltration risk but makes updates slow and capacity fixed; plan for older models and slower patching as accepted costs.

Common mistake

Assuming the gap makes the supply chain safe, and importing pickle checkpoints or unscanned packages because nothing can call out anyway.

Follow-up

A critical vulnerability is announced in the inference server. Walk through getting a patched version running inside the network within 48 hours.

Question 24HardDeveloper tools

Design a code assistant for 4,000 engineers that answers questions about the company's 30,000 repositories, respecting who can see which repository.

#

Repositories range from tiny to a 20-million-line monorepo; there are about 50,000 commits a day. Engineers want answers like "where is rate limiting implemented for the payments API?" in seconds, with links to files and lines.

Show answer

Requirements: answer latency, freshness after a push (for example, under 10 minutes), repository-level permissions mirrored from the Git host, and citations to exact files and lines.

Indexing: - Chunk by code structure (functions, classes, files with headers), keeping the path, symbol names and language with each chunk. - Index lexically (identifiers, strings, paths) and semantically (embeddings of code and docstrings); a symbol graph (definitions, references, imports) helps answer "who calls this". - Incremental updates from push webhooks: diff the commit, re-index only changed files, and process deletes and renames. A periodic reconciliation catches missed events. - Skip vendored and generated code, and scan for secrets before indexing so credentials never become retrievable text.

Query path: - Rewrite the question into lexical and semantic queries; extract symbols mentioned. - Retrieve with the user's repository permissions as a filter inside the search, fuse results, and expand along the symbol graph (definition plus key callers). - The model answers with file and line citations that the UI links to the exact commit.

Permissions: sync repository membership from the Git host, cache per user for a few minutes, and re-check before showing private file content. Private-repository content must never land in a shared cache.

Arithmetic: 50,000 commits a day is about 0.6 per second on average, but pushes cluster in working hours, so size the indexer for several times that.

Evaluation: a set of real questions with known answer locations, scored by whether the right files are retrieved and cited.

Observability: index lag per repository, retrieval hit rate on the eval set, citation-click rate, and "no relevant code found" rate.

Trade-offs

A symbol graph improves "where is this used" answers but costs language-specific tooling; embeddings alone are easier to build but miss exact identifiers and call relationships.

Common mistake

Indexing everything with one service account and filtering results by permission only in the UI.

Follow-up

A repository is made private after being public for a year. What must happen in the index and caches?

Question 25MediumData flywheel

Design the system that turns user feedback on an AI feature into better evaluation sets and model or prompt improvements.

#

A writing assistant with 500,000 weekly users has a thumbs up/down button that about 1% of users press, mostly when unhappy. The team wants to "learn from feedback" but has no pipeline.

Show answer

Requirements: consent and privacy rules for using user content, which signals count, how fast feedback should reach evaluation sets, and who labels.

Signals: - Explicit: thumbs, written comments, reports. - Implicit: the user accepted, edited heavily, regenerated, copied, or abandoned the output. These are noisier, but far more common.

Pipeline: 1. Join each signal to its trace (prompt version, model, retrieved sources, output) by request ID. 2. Filter by consent and privacy policy; redact where needed. 3. Cluster failures by topic and error type to find patterns instead of reading items one by one. 4. Sample for human labelling, stratified by cluster and not only from thumbs-down, since thumbs-down users are a biased sample. 5. Labelled cases go into the regression evaluation set, so a fixed failure stays fixed. 6. Improvements (prompt changes, retrieval fixes, fine-tuning data) go through the normal evaluation gate.

Bias: 1% of 500,000 users is about 5,000 pieces of explicit feedback a week, skewed towards complaints. Never read the thumbs-down rate as the failure rate; estimate quality from a random labelled sample instead.

Model judges can help triage at scale, but validate them against human labels first; the EvalGen work found that people's grading criteria shift as they look at outputs, so expect to revise rubrics.

Observability: feedback rates by feature and version, label throughput and backlog, time from feedback to an eval case, and the share of regression cases that originate from production.

Trade-offs

Implicit signals give volume but are ambiguous; explicit feedback is clearer but rare and biased, so combine both and calibrate with labelled samples.

Common mistake

Treating the thumbs-down rate as the error rate, or training directly on thumbs-up outputs.

Follow-up

How would you measure whether the flywheel actually improves the product quarter over quarter?