supraj.dev

Interview questions & answers

AI platform serving & scaling

Find the serving bottleneck before adding GPUs. Balance latency, throughput and memory.

  • 43 questions
  • 2 easy
  • 23 medium
  • 18 hard
  • 4 related articles

43 questions

Question 1EasyProduction reasoning

A streamed answer starts late but then generates quickly. What would you measure?

#

Users wait several seconds before the first token, while token streaming looks fast.

Show answer

Measure end-to-end time and instrument retrieval, queue wait, prompt processing and token generation. Compare time to first token with time between subsequent tokens. Break results down by input length and concurrency. Fix the dominant measured delay rather than judging only tokens per second.

Trade-offs

Streaming improves perceived responsiveness but does not remove work before the first token.

Common mistake

Using a single average response-time metric for every workload.

Follow-up

What would change if long retrieved contexts caused the delay?

A useful sequence to explain
  1. Retrieval
  2. Queue wait
  3. Prompt processing
  4. First token
  5. Token generation
Question 2MediumProduction reasoning

Throughput improves with larger batches, but chat latency gets worse. Why?

#

A benchmark optimization processes more tokens per second but harms interactive users.

Show answer

Benchmark realistic prompt and output lengths at target concurrency. Inspect queueing, active requests and scheduling configuration. Bound admission and separate interactive traffic from offline jobs when useful. Choose a configuration that meets the latency objective at sustainable throughput, then test overload behavior.

Trade-offs

Larger batches can improve utilization while increasing waiting time and memory pressure.

Common mistake

Optimizing a synthetic maximum-throughput run as if it were the chat workload.

Follow-up

How would you reserve capacity for interactive requests during a batch job?

Question 3MediumProduction reasoning

Prefix caching helped long prompts, but long answers are still slow. What explains it?

#

Many requests share instructions and documents. Their first tokens arrive faster after warming the cache.

Show answer

A reusable prefix KV cache can avoid processing the shared input again; it does not eliminate generation of new output tokens. Compare prefix hit rates, prompt-processing time and decode time separately. Check whether prompts really share the same token prefix and consider tenant isolation before sharing cached state.

Trade-offs

Caching uses memory and depends on reuse; it can help shared prompts without helping output-heavy workloads.

Common mistake

Treating prefix caching as a cache of the final answer.

Follow-up

Which change to context ordering might reduce prefix reuse?

Question 4HardProduction reasoning

The model fits on the GPU, but concurrency triggers out-of-memory errors. What else consumes memory?

#

One request succeeds. Many long-context requests fail under load.

Show answer

Account for runtime allocations and KV state for active sequences, then reproduce with measured input lengths and concurrency. Inspect serving memory limits and capacity metrics. Bound sequence lengths and admission; consider quantization, distribution or additional capacity only after checking quality and workload fit.

Trade-offs

Reducing precision or context can increase capacity but may change quality; adding GPUs introduces cost and communication overhead.

Common mistake

Sizing capacity from model-weight memory alone.

Follow-up

How would you establish a safe concurrency limit without hiding queue growth?

Question 5HardProduction reasoning

CPU is low while the inference queue grows. What should drive scaling?

#

The CPU-based autoscaler does not react to overloaded GPU serving replicas.

Show answer

Inspect queue depth, wait time, active sequences and GPU capacity for the workload. Use a suitable custom or external metric through the autoscaling metrics pipeline. Test model-loading delay, readiness, capacity limits and scale-down behavior. Keep admission controls because autoscaling cannot instantly absorb a burst.

Trade-offs

Queue-based scaling reacts to demand but must account for startup delays and downstream limits.

Common mistake

Assuming CPU utilization describes GPU-serving pressure.

Follow-up

How would you prevent a newly started but unloaded replica from receiving traffic?

Question 6HardServing & operations

What makes production LLM deployment harder than serving a deterministic API?

#
Show answer

Deployment capacity depends on model weights, KV-cache growth, prompt lengths, generated-token rates, batching, concurrency, GPU topology and queueing. Operational quality also depends on time to first token, throughput, error rate, model/provider changes, data boundaries, cost budgets and evaluation of unpredictable outputs.

A production architecture should measure prefill and decode separately, autoscale against meaningful demand signals, bound request/token budgets, rate-limit abusive traffic and define fallback and rollback behavior. Protect prompts and retrieved data according to their sensitivity, and verify quality after each model or prompt release. The objective is a reliable *completed task*, not maximum raw tokens per second at any cost.

Trade-offs

Larger batches increase throughput but can worsen interactive latency and memory pressure; route by workload and SLO.

Common mistake

Buying more GPUs before measuring the queue, prefill/decode stages and memory residency.

Follow-up

If prompt latency increases while GPU memory is high and CPU usage stays low, what signals should control scaling?

Question 7MediumvLLM

How does speculative decoding speed up generation, and when might it disappoint?

#
Show answer

A cheap draft mechanism proposes several tokens, and the target model verifies them together. Accepted tokens reduce sequential target-model work. Drafting can use a smaller model or techniques such as matching n-grams; it does not always require a second model.

Measure accepted tokens per proposal, inter-token latency, throughput and memory on representative traffic. Low acceptance or expensive drafting can erase the gain. Benchmark the actual concurrency and prompt lengths; a published speedup is not a capacity guarantee.

Question 8MediumvLLM

What separate problems do PagedAttention and continuous batching address?

#
Show answer

PagedAttention addresses KV-cache memory management: the original design maps a sequence’s logical blocks to physical blocks instead of reserving one large contiguous region. Continuous batching addresses scheduling: completed requests leave the active work set and new requests can join while other sequences continue generating.

These mechanisms complement each other, but neither guarantees lower latency at every load. Inspect cache pressure, queue wait and generation latency separately. Treat the original PagedAttention paper as a conceptual explanation; current engines may use different attention kernels.

Question 9HardvLLM

When might separating prefill from decode help an inference service?

#
Show answer

Separating prompt processing and token generation lets you tune their capacity and parallelism independently and reduce interference from long prefills during ongoing decode. The service must transfer the correct KV state between compatible instances.

Measure first-token and tail inter-token latency, transfer overhead, failures and utilization against a colocated baseline. Extra replicas, routing and network traffic can outweigh the benefit. vLLM currently labels disaggregated prefilling experimental and explicitly cautions that it does not improve throughput; choose it for a measured latency problem, not an assumed speedup.

Question 10MediumRealtime voice

A user interrupts a voice agent. Why is cancelling generation alone insufficient?

#
Show answer

Stop audio playback as well as the active response, then ensure conversation state reflects what the user actually heard. Otherwise the next turn may refer to an answer that was generated but never played. Preserve any already-executed tool action and reconcile its outcome separately.

OpenAI Realtime’s WebRTC/SIP paths manage output buffering and truncation on interruption. With WebSockets, the client manages playback and must handle truncation of the unplayed response. Test barge-in, network delays and human handoff; do not assume cancellation reverses an external action.

Question 11MediumCaching

How do exact-response, semantic-response and prefix caches differ?

#
Show answer

An exact-response cache reuses a completed answer for the same effective input and configuration. A semantic-response cache reuses an answer for a sufficiently similar request, which risks treating distinct questions as equivalent. Prefix caching reuses input computation; it still generates a new answer.

Include tenant, authorization scope, data version and relevant model/prompt configuration in cache policy. Expire or invalidate stale answers and test negative cases. Prefix-cache hits do not bypass all model execution, and a semantically similar prompt is not permission to return another user’s result.

Question 12MediumvLLM

Time to first token is fine at low load, but throughput collapses at peak. What do you tune?

#

Above 60 concurrent users, latency jumps and requests time out.

Show answer

Check vLLM's metrics for waiting requests, KV cache usage and preemptions. When the KV cache fills, requests wait or are preempted and recomputed, which wastes GPU time.

Tune the limits: max_num_seqs and max_num_batched_tokens control batch size, gpu_memory_utilization controls how much memory the KV cache can use, and a smaller max_model_len leaves room for more requests if you do not need very long contexts. If the cache is truly full, add replicas or use quantization to free memory.

Trade-offs

Bigger batches raise throughput but increase latency per request; tune against your latency target.

Common mistake

Raising batch limits without watching preemptions, which can make throughput worse.

Follow-up

Which metric would tell you to add a replica rather than tune further?

Question 13HardAdapters

Thirty customers each have a LoRA fine-tune of the same base model. How do you serve them without thirty deployments?

#

Each fine-tune gets little traffic, but together they would need thirty GPUs if served separately.

Show answer

Load the base model once and serve the adapters on top of it. Engines such as vLLM can enable LoRA support, keep several adapters in GPU memory at once, and choose the adapter per request, usually by model name, while still batching requests for different adapters together.

Plan for limits: the number of adapters resident at once and their maximum rank, the cost of loading a cold adapter, and some extra compute per request. With many replicas, route requests to replicas that already have the adapter loaded; the Gateway API Inference Extension can use loaded adapters as a routing signal. Evaluate each adapter separately, since they share infrastructure but not quality.

Trade-offs

Shared serving saves GPUs, but one noisy tenant can affect latency for others on the same replica.

Common mistake

Merging every adapter into a separate full model copy.

Follow-up

How would you give one premium customer guaranteed capacity in this setup?

Question 14MediumOptimization

When does quantization help serving, and what must you verify?

#

A 70B model needs four GPUs. Someone suggests a 4-bit version on two.

Show answer

Quantization stores weights, and sometimes activations or KV cache, in fewer bits, such as FP8 or INT4 formats like AWQ or GPTQ. The model takes less memory, so it fits on fewer GPUs or leaves more room for the KV cache and larger batches.

Verify three things: quality on your own evaluation set, not just a public benchmark; that your hardware and serving engine support the format efficiently; and real throughput and latency, since some formats save memory without speeding things up.

Trade-offs

Lower precision saves cost but can hurt quality on reasoning or rare-language tasks more than on general chat.

Common mistake

Shipping a quantized model after a quick manual chat test.

Follow-up

Which tasks in your product would you test first for quality loss?

Question 15HardKV cache

Product wants 128K-token context for a Llama 3 70B chat product on one 8x H100 80GB node. How many full-length conversations can that node hold at once, and what do you tell product?

#

BF16 weights and BF16 KV cache, tensor parallel across all 8 GPUs. Most conversations are short, but a few customers paste whole contracts.

Show answer

Start from the KV cache formula. Each token stores a key and a value vector for every layer and every KV head: bytes per token = 2 x layers x KV heads x head dimension x bytes per element. Llama 3 70B has 80 layers, 8 KV heads (grouped-query attention) and a model dimension of 8,192 over 64 attention heads, so the head dimension is 128. In BF16 that is 2 x 80 x 8 x 128 x 2 = 327,680 bytes, which is 320 KiB per token.

A full 128K (131,072-token) conversation therefore needs 40 GiB of KV cache on its own. The node has 8 x 80 GB = 640 GB of HBM. The weights take about 140 GB (70 billion parameters x 2 bytes), and the runtime also needs room for activations and CUDA graphs. That leaves somewhere around ten full-length conversations for the whole node. Grouped-query attention is already helping: with one KV head per query head (64 instead of 8), each token would need eight times as much.

Tell product that capacity is set by tokens in flight, not by users. vLLM allocates KV blocks as tokens arrive, so a thousand short chats can share the node while a handful of 128K sessions can fill it and force preemption for everyone else. Recommend a per-tier max_model_len, an admission limit for very long requests (or a separate pool for them), and a measured FP8 KV cache, which stores one byte per element instead of two. Then load-test with the real length distribution and watch vllm:kv_cache_usage_perc.

Trade-offs

A long-context tier sells well but each full-length session displaces dozens of short ones; a separate pool keeps the cost visible but runs at lower utilisation.

Common mistake

Sizing GPUs from the weights alone, or from average prompt length, and discovering the KV cache limit at the first big customer.

Follow-up

How would the numbers change for the 8B model, and why is that model much cheaper per long conversation?

Question 16MediumQuantization

You turned on --kv-cache-dtype fp8 to fit more concurrent requests. Throughput improved, but answers on long documents got worse. What went wrong and how do you fix it without giving up the memory saving?

#

No calibration step was run. The model mixes full attention and sliding-window attention layers.

Show answer

An FP8 KV cache stores keys and values in 8 bits, so the same memory holds more tokens. The quality of that depends on the scales used to map values into FP8. vLLM's quantized KV cache documentation says that without calibration all quantization scales are set to 1.0. Values outside the FP8 range then lose precision, and the error accumulates over long sequences because every new token attends over the whole stored history.

Fix it in three steps. First, calibrate: vLLM recommends estimating scales with LLM Compressor on a calibration dataset, and supports per-tensor scales (and per-attention-head scales with the Flash Attention backend). Second, the documentation notes that some layer types, such as sliding-window attention, are more sensitive to KV cache quantization; --kv-cache-dtype-skip-layers keeps those layers in the native data type. Third, evaluate on long inputs specifically, because a short-prompt benchmark will not show this damage.

Also know which format you picked. fp8_e4m3 has more precision and less range, fp8_e5m2 the opposite. And with the Flash Attention 3 backend the queries are quantized too, so attention itself runs in FP8.

Trade-offs

An FP8 KV cache roughly doubles the tokens that fit compared with BF16, but it needs calibration and long-context evals; skipping sensitive layers gives back some of the memory.

Common mistake

Judging a KV cache quantization change with short prompts, where the accumulated error never shows up.

Follow-up

How would you build a long-context regression eval that runs in CI before a serving config change?

Question 17MediumScheduling

Streaming chat stutters for everyone whenever one user pastes a 60-page document. How do you reduce the stutter, and what do you give up?

#

vLLM V1, chunked prefill at its default, mixed chat and document traffic on the same replicas.

Show answer

The stutter is prefill-decode interference. A long prompt must be processed (prefilled) before its first token, and while that work runs in a step, the decodes of other users in the same batch wait. In vLLM V1, chunked prefill is enabled by default whenever possible: a long prompt is split into chunks so that it shares each step with running decodes instead of blocking them.

The tuning knob is max_num_batched_tokens, the token budget per step. vLLM's tuning guide says smaller values (for example 2048) give better inter-token latency because fewer prefill tokens slow down decodes, while higher values give better time to first token because more prompt tokens are processed per step. It recommends values above 8192 for smaller models on large GPUs, so the right number depends on which latency you promised.

Measure with the histograms vLLM exposes: vllm:inter_token_latency_seconds for the stutter and vllm:time_to_first_token_seconds for the cost to the document user, both at p99 and broken down by prompt length. If no single budget meets both SLOs, separate the traffic: route long-document requests to their own pool, or move to disaggregated prefill and decode, which DistServe showed removes this interference by placing the two phases on different GPUs.

Trade-offs

A smaller step budget protects streaming users at the cost of slower first tokens for long prompts; separate pools protect both but need more capacity planning.

Common mistake

Raising the step budget to improve a TTFT dashboard and making every streaming user stutter.

Follow-up

How would you decide between a separate long-prompt pool and full prefill/decode disaggregation?

Question 18HardParallelism

You have one node with 8x H100 80GB for a 70B model. Do you run one replica with tensor parallel 8, two replicas with tensor parallel 4, or something else? How do you decide?

#

BF16 weights of about 140 GB. Traffic is interactive chat with a p99 inter-token latency target. A second node may arrive next quarter.

Show answer

First check what fits. 140 GB of weights split over 4 GPUs is 35 GB per GPU, leaving roughly 45 GB per GPU for KV cache and runtime. Over 8 GPUs it is 17.5 GB per GPU, so one TP 8 replica has more total KV room than either TP 4 replica, but the two TP 4 replicas together have about the same total.

Then weigh the costs. vLLM's guide describes tensor parallelism as sharding each layer across GPUs, the common choice inside a single node, and warns that a higher tensor parallel size may cause excessive synchronisation overhead. Data parallelism replicates the whole model and is the tool for more throughput once a replica fits. So two TP 4 replicas usually give more aggregate throughput and an independent failure domain, while TP 8 gives each request more compute, which can lower per-token latency, and more KV room for very long sequences.

For the second node, keep tensor parallelism inside the node over NVLink, and use pipeline parallelism (or data parallelism) between nodes. vLLM recommends pipeline parallelism when you have maxed out efficient tensor parallelism or need to go across nodes, and warns that it may add latency.

Decide by benchmark, not by rule: run both layouts with your real prompt and output lengths at your target request rate, and compare goodput, the requests per second that meet your latency SLO.

Trade-offs

Bigger tensor parallel groups trade synchronisation cost for per-request speed and KV room; more replicas trade KV room for throughput and fault isolation.

Common mistake

Stretching tensor parallelism across nodes over the network instead of keeping it on NVLink.

Follow-up

How would expert parallelism change this plan for a mixture-of-experts model of similar total size?

Question 19HardParallelism

You are deploying a large mixture-of-experts model on vLLM across several GPUs. How do you shard it, how do you explain its GPU bill to finance, and what do you watch once it is live?

#

The team plans to serve it on vLLM across several GPUs.

Show answer

Active parameters set the compute per token; total parameters set the memory. A router picks a few experts per token, so the FLOPs per token look like a much smaller dense model. But any token can be routed to any expert, so every expert must be resident in GPU memory. The memory bill follows the total size, and the KV cache comes on top of that.

That is why the GPU count looks like a dense model's. The win is elsewhere: for the same memory, you get the quality of a large model at the compute cost of a small one, which helps throughput and latency once the weights fit.

For sharding, vLLM supports expert parallelism for MoE models, enabled with enable_expert_parallel=True, which places different experts on different GPUs instead of splitting every expert. The expert parallel degree is the same as the tensor parallel degree, and MoE layers are sharded over the product of the tensor parallel and data parallel sizes. That lets attention layers use data parallelism while experts are spread across the group.

Watch for imbalance: if routing sends most tokens to a few hot experts, the GPUs holding them become the bottleneck. Measure per-GPU utilisation under real traffic, not only total throughput.

Trade-offs

Expert parallelism avoids splitting every expert matrix but adds all-to-all traffic and is sensitive to routing imbalance.

Common mistake

Sizing MoE hardware from active parameters and running out of memory at load time.

Follow-up

How would you detect expert hot spots in production, and what can you change if you find them?

Question 20HardSecurity

Security review of your multi-tenant inference platform asks whether one customer can learn another customer's prompt through the shared prefix cache. What is the risk and what do you change?

#

Automatic prefix caching is on. Several customers share replicas. Some prompts contain confidential case notes.

Show answer

The risk is a timing side channel. With prefix caching, a request whose prompt prefix is already in the cache skips that prefill work, so its first token arrives measurably faster. An attacker who can guess candidate prefixes can send them and watch time to first token: a fast response suggests that someone else recently sent that text.

vLLM addresses this with a cache_salt field on the request. The salt is mixed into the hash of the first block, so only requests carrying the same salt can reuse each other's cached KV blocks. vLLM's design notes describe it as protection against timing-based attacks, with sharing limited to users or requests that agree on a common salt. Set the salt per tenant (or per user for the most sensitive data) in your gateway, never from client input.

Also check the hash function. vLLM's --prefix-caching-hash-algo defaults to sha256; the xxhash options are faster but non-cryptographic, and the documentation warns that collisions could cause undefined behaviour or leak private information in multi-tenant environments. Keep a cryptographic hash on shared clusters.

The trade-off is hit rate: per-tenant salts mean tenants no longer share cached system prompts. That cost is usually small, because each tenant still reuses its own prefixes.

Trade-offs

Per-tenant salts close the side channel but remove cross-tenant cache sharing; a fast non-cryptographic hash saves CPU but adds collision risk.

Common mistake

Treating the prefix cache as an internal performance detail with no security boundary.

Follow-up

Which other shared serving components could leak information between tenants through timing or errors?

Question 21MediumAutoscaling

Your GPU autoscaler adds a vLLM replica during a traffic spike, but the replica takes so long to become ready that the spike is over. Where does the time go and how do you shorten it?

#

Large model weights pulled from object storage at start-up, torch.compile and CUDA graphs enabled, readiness gated on the model being loaded.

Show answer

Break start-up into phases and time each one: node provisioning and image pull, weight download, weight loading onto the GPUs, memory profiling, torch.compile, and CUDA graph capture. Fix the largest first.

For weights, vLLM can load with --load-format runai_streamer, which reads tensors concurrently and streams them to GPU memory; it supports S3, Google Cloud Storage, Azure Blob Storage and S3-compatible stores, with a concurrency setting passed through --model-loader-extra-config. The documented format is Safetensors.

For compilation, vLLM persists torch.compile artifacts under VLLM_CACHE_ROOT (default ~/.cache/vllm). Put that cache on a volume that survives pod restarts, and set VLLM_FORCE_AOT_LOAD=1 in CI or staging so a cache miss is an error instead of a silent recompile. Passing the logged --kv-cache-memory value back skips the memory profiling and CUDA graph memory estimation pass.

There is also a direct trade-off: --enforce-eager (or -O0) skips compilation and graph capture for the fastest start-up, at the cost of steady-state decode speed. That can be right for a burst-only pool. Finally, accept that GPU scale-up is slow: keep a warm minimum, scale on queue signals early, and keep admission control so a burst queues instead of failing.

Trade-offs

Faster start-up from eager mode costs steady-state speed; warm spare replicas cost money every hour.

Common mistake

Treating an inference pod like a stateless web pod and expecting it to scale up in seconds.

Follow-up

How would you pre-stage weights on new GPU nodes before any pod is scheduled?

Question 22EasyPerformance

A 4-GPU vLLM pod runs far below expected throughput. GPU utilisation is low and spiky, and the pod spec sets a CPU limit of 2. Is that related?

#

Tensor parallel 4. The CPU limit was copied from a small web service template.

Show answer

Yes. vLLM runs host processes to drive each GPU as well as the API server and engine core. Its tuning guide warns that using fewer physical CPU cores than processes causes contention and significantly degrades throughput and latency, and states a minimum of 2 + N physical cores for N GPUs. A 4-GPU pod therefore needs at least 6 physical cores, and a limit of 2 starves the scheduler that keeps the GPUs fed.

In Kubernetes the CPU limit also adds CFS throttling, which shows up as stalls at irregular intervals: the spiky utilisation you see. Give the pod a CPU request that meets the minimum (whole cores if you use the static CPU manager), drop or raise the CPU limit, and check container_cpu_cfs_throttled_periods_total afterwards.

The broader lesson is that GPU pods are not GPU-only. Tokenisation, detokenisation, scheduling, structured output grammars and the HTTP server all run on the CPU.

Trade-offs

Reserving more CPU per GPU node lowers how many other pods fit there, but GPU time is far more expensive than CPU time.

Common mistake

Copying resource templates from web services to inference pods.

Follow-up

Which other host resources (shared memory, NUMA placement, PCIe topology) can bottleneck a multi-GPU pod?

Question 23HardPerformance

One user streaming from an 8B model on an H100 gets about the same tokens per second whether you buy a faster-compute GPU or not, but serving 64 users at once costs little extra per user. Explain with numbers.

#

Llama 3 8B in BF16 on one H100 SXM. The team expected token speed to scale with advertised TFLOPS.

Show answer

Decoding one token for one sequence needs every weight once. 8 billion parameters in BF16 is about 16 GB, and the H100 SXM's memory bandwidth is 3.35 TB/s. Just reading the weights takes roughly 16 / 3,350 s, about 4.8 ms, so a single stream cannot exceed roughly 200 tokens per second however many FLOPS the GPU has. Decode at small batch is memory-bandwidth bound; the tensor cores mostly wait.

Batching changes the arithmetic. The same weight read serves every sequence in the batch, so 64 users share one pass over the weights. The work per pass grows (each sequence also reads its own KV cache), but the dominant weight traffic is amortised. That is why continuous batching raises throughput so much, and why cost per token falls with concurrency until KV cache memory or compute becomes the limit.

Prefill is the opposite: it processes many prompt tokens per weight read, so it is compute bound. This split explains several serving choices. Weight-only quantization (fewer bytes to read) helps small-batch decode most. Speculative decoding helps because the target model verifies several tokens per weight read, and vLLM documents it as aimed at medium-to-low QPS, memory-bound workloads. Faster compute mostly helps prefill and large batches.

Note that NVIDIA's headline tensor-core figures are quoted with sparsity, so do not use them as a dense compute budget.

Trade-offs

Bigger batches lower cost per token but raise per-user latency; quantization and speculation cut latency but need quality checks.

Common mistake

Choosing inference GPUs by peak TFLOPS when the workload is bandwidth bound.

Follow-up

At roughly what batch size would you expect decode to stop being bandwidth bound, and how would you measure it?

Question 24MediumBenchmarking

A vendor benchmark shows a tokens-per-second figure twice what you measured on the same GPU and engine. Before arguing, how do you design a benchmark that predicts production?

#

You used vllm bench serve with default settings. Production traffic is bursty chat with long system prompts.

Show answer

Most of the gap is usually method. vllm bench serve defaults are built for stress, not realism: --request-rate defaults to inf, which sends all requests at time zero; the default dataset is random with --random-input-len 1024 and --random-output-len 128. Throughput numbers from that setup say little about a chat product.

Make it production-shaped. Use a dataset close to your traffic (sharegpt, custom, or prefix_repetition when long shared prompts matter, because they change prefix cache behaviour). Set a finite --request-rate and use --burstiness below 1.0 for burstier arrivals, or cap in-flight work with --max-concurrency. Decide whether --ignore-eos is appropriate: it forces full-length outputs, which is good for repeatability but unlike real chat.

Report latency percentiles, not averages: --percentile-metrics covers ttft, tpot, itl and e2el, and --metric-percentiles defaults to 99. Most useful is --goodput, which takes SLO pairs in milliseconds such as ttft:500 tpot:50 and counts only requests that meet them. Sweep the request rate and plot goodput: the knee of that curve is your real capacity per replica.

Then compare like with like: same model revision, quantization, tensor parallel size, engine version and sampling settings as the vendor.

Trade-offs

Synthetic random prompts are repeatable but unrealistic; replaying real traffic is realistic but needs privacy handling and changes as traffic changes.

Common mistake

Quoting peak throughput at infinite request rate as the capacity of a latency-sensitive service.

Follow-up

How would you turn the goodput curve into a replica count and an autoscaling threshold?

A production-shaped sweep point with an SLO · shell
vllm bench serve --model my-model \
  --dataset-name sharegpt --dataset-path ./traffic_sample.json \
  --request-rate 8 --burstiness 0.5 \
  --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,90,99 \
  --goodput ttft:500 tpot:50
Question 25MediumSLOs

You must write latency SLOs for three LLM products: an interactive chat, a voice agent, and a nightly document summariser. Which metrics do you choose for each, and why not just "average latency"?

#

All three run on the same inference platform; the platform team is asked for one dashboard.

Show answer

Use the metrics that match what the user experiences. Time to first token (TTFT) is the wait before anything appears. Time per output token (TPOT) is the average gap after that, and inter-token latency (ITL) is the distribution of individual gaps, which catches stutter that an average hides. End-to-end latency is total time. vLLM exposes these as histograms: vllm:time_to_first_token_seconds, vllm:request_time_per_output_token_seconds, vllm:inter_token_latency_seconds and vllm:e2e_request_latency_seconds, plus vllm:request_queue_time_seconds for time spent waiting.

Chat: p99 TTFT and p99 ITL, because people notice a slow start and a stuttering stream. Voice: a tighter TTFT, and ITL high enough to stay ahead of speech playback, since a gap becomes audible silence. The summariser: end-to-end completion within a batch window plus throughput per GPU-hour; per-token latency hardly matters.

Average latency mixes short and long outputs and hides tails. A better platform-level measure is goodput, which DistServe frames as the request rate served while meeting both TTFT and TPOT targets. Measure SLOs at the gateway as well as at the engine, because queueing in front of the engine is invisible to engine metrics.

Trade-offs

Tight per-token SLOs force smaller batches and more GPUs; looser ones raise utilisation. Separate SLO classes let one platform run both.

Common mistake

Publishing a single average-latency SLO across all request lengths and products.

Follow-up

How would you translate these SLOs into scheduling classes or separate pools?

Question 26MediumCost

Finance wants to know whether self-hosting an open-weights model is cheaper than a hosted API. How do you calculate your real cost per million tokens, and what usually makes the first estimate wrong?

#

You have benchmark results for one replica and an hourly price for the GPU node. Traffic is high in business hours and low at night.

Show answer

The basic formula is cost per hour of a replica divided by tokens served per hour by that replica. The mistakes are in the denominator.

First, use goodput, not peak throughput: the tokens per hour you can serve while meeting your latency SLO. A benchmark at infinite request rate can overstate it several times over. Second, use realised utilisation. If you keep capacity for peak and traffic falls at night, your average tokens per hour are much lower than the benchmark. Example arithmetic: a replica that costs C per hour and serves T tokens per hour at full SLO load costs C / T per token at peak, but at 35% average utilisation it costs C / (0.35 x T), almost three times as much.

Third, separate input and output tokens. Prefill is compute bound and processes many tokens per step, decode is generated one step per token, so output tokens cost far more GPU time than input tokens. Your traffic mix (long prompts with short answers, or the reverse) changes the result, which is also why hosted APIs price them differently. Fourth, add the costs that are not GPU time: engineers on call, idle warm replicas, evaluation runs, failed requests, storage and egress.

The recommendation is usually mixed: self-host steady, high-volume, latency-tolerant workloads where you can run hot, and keep spiky or low-volume workloads on an API.

Trade-offs

Self-hosting gives control over data, latency and model choice but turns a variable bill into fixed capacity you must keep busy.

Common mistake

Dividing the node price by a peak benchmark number and calling that the cost per token.

Follow-up

Which levers (quantization, batching, prefix caching, scheduling batch jobs at night) improve the denominator most for your traffic?

Question 27MediumCorrectness

Your regression tests send the same prompt at temperature 0 and compare outputs, but they fail randomly in staging under load and pass when run alone. Why, and what do you do?

#

vLLM serving, staging shared with other test traffic. The test asserts exact string equality.

Show answer

Greedy decoding is only deterministic if the logits are bit-identical, and on a busy server they are not. Thinking Machines' analysis argues the usual explanation (concurrency plus floating point) is incomplete: the real cause is that common kernels are not batch invariant. Their numbers for a given input change with the batch size, and the batch size depends on whatever other traffic is on the server. In their experiment, 1,000 temperature-0 runs of one prompt on Qwen3-235B produced 80 unique completions; with batch-invariant kernels all 1,000 matched.

vLLM offers a batch invariance mode, enabled with VLLM_BATCH_INVARIANT=1. Its documentation marks it as beta, lists hardware requirements, and says it may cost performance compared with the default mode. Speculative decoding adds its own caveat: vLLM does not guarantee stable logprobs with it.

For tests, do not assert exact strings from a shared server. Assert properties (valid JSON, required fields, a correct answer judged by a rubric or a checker) and score across several samples. If you truly need reproducibility, for example for audits or debugging, run a dedicated deployment with batch invariance enabled and pin the model revision, engine version and sampling settings.

Trade-offs

Batch invariance buys reproducibility at a throughput cost; property-based checks are robust but need more design than string comparison.

Common mistake

Treating temperature 0 as a guarantee of identical outputs on a shared server.

Follow-up

How would you design an eval that tolerates harmless variation but still catches a real regression?

Question 28MediumStructured output

After a vLLM upgrade, a service that relied on guided_json starts returning free text that breaks the JSON parser downstream. What changed, and how do you make structured output robust to the next upgrade?

#

The client passes guided_json in extra_body and never validated responses because "the engine guarantees it".

Show answer

vLLM's structured outputs documentation says the guided_json field and the other guided_* fields were removed in v0.12.0, along with guided_decoding_backend. The constraint now goes in structured_outputs (for example {"structured_outputs": {"json": schema}} in extra_body), or you use response_format with type json_schema in the OpenAI-compatible API. A client still sending the old field gets an unconstrained answer.

Migrate to the supported parameter, and pin the engine version so this kind of change arrives through a tested upgrade, not a rebuilt image. Note the backend details too: the default backend is auto, which picks xgrammar or guidance per request, and the documentation warns that regex syntax differs between backends and that changing backends can change the allowed tokens and therefore the output.

Then remove the assumption that caused the outage. Validate every response against the schema in your service, and treat a failure as a handled error (retry, repair or fall back) with a metric on it. Add a contract test to the engine upgrade pipeline that sends each schema your services use and checks the result parses.

Trade-offs

Constrained decoding makes valid output very likely but costs some CPU and limits schema features; validation in the service costs little and catches everything else.

Common mistake

Trusting the serving engine to enforce a contract the client never checks.

Follow-up

When would you choose constrained decoding over tool calling or a post-hoc repair step?

Question 29MediumScheduling

A nightly backfill job sends thousands of summarisation requests to the same vLLM pool as live chat, and chat latency spikes. What scheduling options do you have, and which do you pick?

#

One shared pool to save GPUs. The backfill has no deadline tighter than morning.

Show answer

By default vLLM schedules first come, first served (--scheduling-policy fcfs), so a flood of batch requests ahead of a chat request makes it wait. vLLM also supports --scheduling-policy priority, where each request carries a priority, lower values are handled earlier, and ties are broken by arrival time. Setting chat to a better priority than backfill lets chat jump the queue.

Priority only changes ordering. It does not create KV cache space, and a large backfill can still fill memory with running sequences. So add admission control in front of the engine: cap concurrent backfill requests (a token-aware rate limit is better than a request count) and let the job queue in your own system, where you can watch it.

The cleanest option, when you can afford it, is separation: run the backfill through a separate pool or an offline batch path, scaled up at night. Shared pools save GPUs only if the noisy tenant is controlled.

Trade-offs

Priority scheduling is cheap but does not bound memory use; separate pools isolate perfectly but leave capacity idle part of the day.

Common mistake

Relying on priority alone and still letting the batch job occupy all KV cache.

Follow-up

How would you enforce fair use between many tenants that all claim interactive priority?

Question 30MediumCorrectness

You moved a model from one serving setup to vLLM and pinned the same prompts, but answers are longer and more random than before. Your client never sets temperature. What do you check?

#

The client sends only messages and max_tokens. The model repository ships a generation config file.

Show answer

Look for whose defaults are filling the gaps. vLLM's --generation-config defaults to auto, which loads the generation config from the model path, so sampling defaults published with the model can apply to any request that does not set them. Your old server may have used different defaults. A model revision bump can change them again without any code change on your side.

Make sampling explicit. Set temperature, top-p, max tokens and stop sequences in every request (or in one client library everyone uses), so behaviour does not depend on the server. If you need server-side defaults, point --generation-config at a folder you control or use --override-generation-config, and keep it in version control with the deployment.

Also check the chat template, since the model's template decides how messages become tokens. A different template between servers changes outputs even with identical sampling. Pin the model revision, compare the rendered prompt token IDs between old and new setups for a few requests, and run the eval set before switching traffic.

Trade-offs

Server-side defaults keep clients simple but hide behaviour in deployment config; explicit client parameters are verbose but portable.

Common mistake

Comparing two serving stacks with prompts pinned but sampling parameters and chat templates left to defaults.

Follow-up

How would you detect behaviour drift caused by a model revision bump before users do?

Question 31MediumKV cache

Agent sessions run many turns over long histories, with pauses for tool calls. Between turns their prefix cache is often evicted, so each turn pays full prefill again. What can you do?

#

GPU KV cache is regularly full at peak; hosts have plenty of spare RAM.

Show answer

The prefix cache keeps freed blocks only until GPU memory is needed again; vLLM evicts least recently used blocks first. A paused agent session is exactly the kind of prefix that goes cold and gets evicted while other requests run.

Add a second tier. vLLM can offload KV cache to CPU memory: --kv-offloading-size sets the size in GiB (it is off by default, and with tensor parallelism the size is summed across ranks), and --kv-offloading-backend chooses native or lmcache. When the session resumes, its blocks come back over PCIe instead of being recomputed. NVIDIA Dynamo's KV block manager extends the idea across GPU, CPU, SSD and remote storage.

Check that it pays. Transferring blocks back is only a win when it is faster than recomputing the prefill, which is true for long prefixes and less so for short ones. Measure TTFT for resumed turns and the prefix hit rate before and after, and size host RAM for it so you do not starve other processes. Pair it with cache-aware routing (see the AI on Kubernetes track) so the resumed turn goes back to the replica (or tier) that holds its blocks.

Trade-offs

Offloading trades host memory and PCIe bandwidth for less recomputation; it adds a component and failure mode to the serving path.

Common mistake

Adding offloading without routing affinity, so resumed turns land on replicas that never saw the session.

Follow-up

How would you decide how long a paused session's KV cache is worth keeping?

Question 32MediumReliability

Every rollout or scale-down of your inference Deployment cuts off some streaming responses mid-answer. Agent runs that take several minutes fail most. How do you drain inference pods safely?

#

Kubernetes default termination grace period, readiness probe on the model server, load balancer in front.

Show answer

When a pod is deleted, Kubernetes runs any preStop hook, sends SIGTERM, waits for the termination grace period (30 seconds by default) and then kills the container. A response that streams for minutes cannot finish in 30 seconds, so it is cut off.

Drain in order. First stop new work reaching the pod: the endpoint is removed from Service endpoints once the pod is terminating, but load balancers and gateways take time to notice, so a short preStop sleep helps avoid new requests arriving during shutdown. Then let in-flight streams finish: set terminationGracePeriodSeconds to cover your longest allowed generation plus margin, and make sure the server finishes running requests on SIGTERM instead of exiting at once. Watch vllm:num_requests_running reach zero in a test.

Cap generation length so the grace period has a known upper bound, and use a PodDisruptionBudget so voluntary disruptions take replicas one at a time. For multi-minute agent work, do not rely on one HTTP stream at all: run it as a job with checkpoints and let the client reconnect to results, so a pod loss costs a retry of one step, not the whole run.

Trade-offs

Long grace periods make rollouts and node drains slow; short ones lose work. Job-style agents cost more design but survive any pod loss.

Common mistake

Using web-service termination defaults for a workload whose requests last minutes.

Follow-up

How would you handle a spot or preemptible GPU node that gives you only a short warning?

Question 33HardArchitecture

Your platform team must choose a serving stack for several open-weights models: vLLM, SGLang, TensorRT-LLM, or NVIDIA Dynamo. How do you frame the decision?

#

NVIDIA GPUs, Kubernetes, a mix of chat, agents with long shared prompts, and batch jobs. Two platform engineers.

Show answer

First sort the options by layer. vLLM, SGLang and TensorRT-LLM are inference engines: they run the model, manage the KV cache and batch requests. NVIDIA Dynamo describes itself as the orchestration layer above inference engines, supporting SGLang, TensorRT-LLM and vLLM rather than replacing them, and adds disaggregated prefill and decode pools, routing on worker load and KV cache overlap, KV cache offloading across GPU, CPU, SSD and remote storage, and an SLA-driven planner. So the real choices are: which engine, and do you need a distributed layer above it yet.

For the engine, judge on your workload, not published claims. SGLang's paper introduced RadixAttention for KV cache reuse and reported up to 6.4x higher throughput than other systems on its benchmark programs, which is relevant for agents with heavy shared prefixes but not a general ranking. TensorRT-LLM is NVIDIA's open-source library for optimised inference, architected on PyTorch, so it follows NVIDIA hardware and formats closely. vLLM runs on NVIDIA, AMD and Intel accelerators and supports many model families.

With two engineers, operational cost decides as much as speed: model support on release day, upgrade cadence, metrics, and how often configs break. A sensible path is one engine for most models, a benchmark harness that can run any engine on your traffic, and a routing layer (Gateway API Inference Extension or Dynamo) only when replica counts make cache-aware routing or disaggregation pay.

Trade-offs

A single engine keeps operations simple but may leave performance on the table for some models; mixing engines optimises each model but multiplies upgrades and runbooks.

Common mistake

Choosing an engine from a vendor benchmark chart instead of a benchmark of your own traffic and SLOs.

Follow-up

What would make you add disaggregated prefill and decode, and how would you prove it paid off?

Question 34MediumQuantization

A team quantized a model to FP8 and saw a large speed-up on their H100 test box, but almost none after deploying to the A100 fleet. Why?

#

Same checkpoint, same vLLM version. The A100 deployment logs a Marlin kernel in use.

Show answer

FP8 matrix math needs FP8 tensor cores. vLLM's FP8 documentation says FP8 computation is supported on NVIDIA GPUs with compute capability 8.9 or higher (Ada Lovelace, Hopper, Blackwell). The A100 is Ampere, below that line, so vLLM runs the checkpoint as weight-only FP8 (W8A16) with Marlin kernels: weights are stored in 8 bits and converted for computation in 16 bits.

That still halves weight memory, which leaves more room for KV cache and helps memory-bound decode, but it cannot give the compute speed-up of true W8A8 FP8, which is what the H100 test measured. vLLM's own figures for FP8 are up to 2x lower model memory and up to 1.6x higher throughput, with minimal accuracy impact; treat those as an upper bound for supported hardware.

Also check how the model was quantized. The recommended scheme quantizes weights statically per channel and activations dynamically per token. Online dynamic quantization at load time is simpler, but the documentation notes its latency improvements are limited. Benchmark each target GPU type separately and keep the quantization choice in the per-fleet config.

Trade-offs

Weight-only quantization on older GPUs still buys memory and batch room but not compute; buying newer GPUs buys both at higher cost.

Common mistake

Generalising a benchmark from one GPU generation to a fleet of another.

Follow-up

For an A100 fleet, when would INT4 weight-only quantization be a better choice than FP8 weights?

Question 35HardSpeculative decoding

You serve three workloads on vLLM: an open-ended chat assistant, a code-editing tool that rewrites files with small changes, and a RAG assistant that quotes retrieved text. Which speculative decoding method would you try for each, and what would make you turn it off?

#

All three run the same 70B-class model. Chat is busy in the daytime; the code tool is used in bursts; latency per user matters more than total throughput.

Show answer

Speculative decoding proposes several tokens cheaply and has the target model verify them in one forward pass. vLLM's documentation states it is theoretically lossless up to the precision limits of hardware numerics, so the choice is about speed, not quality. What differs is where the proposals come from. You set the method in --speculative-config, for example {"method": "eagle3", "num_speculative_tokens": 3}.

- Chat: the next tokens are new text, so you need a model to guess them. vLLM's guide says model-based methods (EAGLE, MTP, draft models) give the best latency reduction. Use MTP if the model was trained with multi-token prediction heads, otherwise an EAGLE head trained for that model, otherwise a small draft model from the same family and tokenizer. - Code editing: most of the output copies the input file with small changes. N-gram (prompt lookup) or suffix decoding finds the continuation in text already seen, needs no extra model, and accepts long runs on copy-heavy output. - RAG quoting: the same idea applies to passages copied from retrieved chunks. Start with n-gram or suffix decoding, and compare it with the model-based method on real traffic.

When to turn it off or scale it back: vLLM's own table rates the gains as lower at high QPS than at low QPS, and the guide describes the technique as aimed at medium-to-low QPS workloads. A busy server is already using its compute on large batches, so verifying rejected guesses costs real work. Also note that vLLM does not guarantee stable token log probabilities, so features that read logprobs need their own check, and that older vLLM versions could not combine speculative decoding with pipeline parallelism (the docs say vllm<=0.15.0).

Decide per workload with a benchmark at your real request rate, measuring inter-token latency and goodput with and without speculation.

Trade-offs

Model-based proposers give the biggest gains on open-ended text but need a matching draft head and GPU memory; n-gram and suffix methods are free to add but only help when output repeats existing text.

Common mistake

Picking one method for every workload from a benchmark run at low load, then finding that it slows the busiest pool at peak.

Follow-up

How would you train or obtain an EAGLE head for a fine-tuned model that has no published draft?

Question 36HardSpeculative decoding

After enabling speculative decoding with 5 speculative tokens, overnight latency improved a lot, but at daytime peak, throughput fell and p99 inter-token latency got worse than without it. How do you diagnose and tune it?

#

vLLM with an EAGLE proposer. Daytime traffic has many concurrent users and more varied prompts, including several languages.

Show answer

Two effects combine.

First, acceptance. Every proposed token is verified, but only the accepted ones save time. Acceptance usually falls with draft position: the first guess is often right, the fifth much less so. vLLM exports vllm:spec_decode_num_accepted_tokens_per_pos and vllm:spec_decode_num_verified_tokens_per_pos; dividing them per position over a time window shows where the guesses stop paying. If daytime traffic (other languages, different domains) differs from what the draft head was trained on, acceptance drops across the board.

Second, load. At low concurrency, decode is limited by memory bandwidth and the GPU has spare compute, so verifying extra tokens is almost free. At peak, large batches already use that compute, so every rejected token is wasted work that delays other users. vLLM's documentation rates the gains of every method as lower at high QPS and describes speculative decoding as aimed at medium-to-low QPS.

Tuning steps: - Reduce num_speculative_tokens to the position where acceptance falls off (often 2 or 3 instead of 5). - Try the dynamic or adaptive variants vLLM lists, which its table rates higher than the base methods at high QPS. - If that is not enough, route peak traffic to a pool without speculation, or keep it only for latency-sensitive tiers. - Retrain or choose a draft that matches the traffic mix if acceptance is low on specific languages or domains.

Verify with vllm bench serve at both the overnight and peak request rates, comparing goodput against your inter-token latency SLO, not tokens per second at one load level.

Trade-offs

More speculative tokens give bigger wins when acceptance is high and the GPU is idle, but waste compute and hurt everyone at peak; fewer tokens give smaller but steadier gains.

Common mistake

Measuring the speed-up at one concurrency level and leaving the setting fixed for all traffic.

Follow-up

How would you alert on a draft model that has drifted away from the target model after a fine-tune?

Question 37HardQuantization

You must pick quantization schemes for one model served in two places: a low-traffic interactive assistant on a few A100s, and a high-throughput batch summarisation pool on H100s. Would you use INT4 weight-only (AWQ or GPTQ), INT8 W8A8, or FP8 W8A8, and why?

#

BF16 model of about 30B parameters. You can run calibration with LLM Compressor. Quality must stay within your eval tolerance.

Show answer

Match the scheme to the bottleneck and to the hardware.

Interactive, low traffic, A100: small batches make decode memory-bound, so the win comes from reading fewer weight bytes per token. 4-bit weight-only (W4A16) does that and frees memory for KV cache. The LLM Compressor guide describes W4A16 as giving useful speed-ups in low-QPS regimes, recommended for any GPU, and says it needs non-round-to-nearest methods such as GPTQ or AWQ, which use a calibration dataset. GPTQ (arXiv 2210.17323) uses approximate second-order information in one pass; AWQ (arXiv 2306.00978) protects the most important weights, found from activation statistics, by scaling channels. Both are well supported on Ampere in vLLM.

High-throughput batch, H100: large batches make the matrix multiplications compute-bound, so you want low-precision math, not just smaller weights. The guide describes FP8 W8A8 as optimised for server and batch inference (and needing no calibration data with round-to-nearest), and INT8 W8A8 for server, batch and high-QPS serving. vLLM's hardware table lists FP8 W8A8 for Ada and Hopper only, and INT8 W8A8 from Turing onward. So FP8 for the H100 pool; INT8 W8A8 if you ever need one 8-bit compute scheme across both fleets.

Two traps. vLLM can fall back silently: when activation-quantizing kernels are unavailable, its docs say W8A8 runs as W8A16 and W4A4 as W4A16, which you can turn into an error with VLLM_STRICT_QUANT_SCHEME=1. And NVFP4, which has full FP4 acceleration on Blackwell, falls back to weight-only kernels on older GPUs.

Finally, evaluate every quantized checkpoint on your own task evals, including long inputs, and benchmark at each pool's real concurrency.

Trade-offs

W4A16 gives the most memory saving and helps small batches, but can lose more quality and does little for compute-bound batches; W8A8 keeps quality closer to BF16 and speeds up large batches, but only on GPUs with the matching kernels.

Common mistake

Choosing one scheme from a single benchmark and deploying it on hardware where vLLM quietly runs it as weight-only.

Follow-up

How would you build a quantization acceptance test that a new checkpoint must pass before it reaches production?

Question 38HardDisaggregation

Leadership read that prefill/decode disaggregation is how large labs serve models, and asks you to move to it "to double throughput". What does disaggregation actually buy you, what does it cost, and how would you decide whether to adopt it?

#

vLLM on two 8-GPU nodes. Prompts are long (RAG with 10K-token contexts) and outputs are short. The inter-token latency SLO is missed at p99 but the median is fine.

Show answer

Correct the goal first. vLLM's documentation says plainly that disaggregated prefill does not improve throughput. What it gives you is control: prefill and decode run in different vLLM instances, so you can tune time to first token and inter-token latency separately, with different parallelism for each, and you can control tail inter-token latency. Without it, vLLM may insert prefill work while other requests are decoding, which raises tail latency. DistServe (arXiv 2401.09670) made the same argument in terms of goodput: requests per second that meet both latency targets.

Your symptoms fit that case: long prompts, p99 inter-token latency missed while the median is fine. That is prefill interfering with decode.

What it costs: - The KV cache for each prompt must move from a prefill instance to a decode instance through a connector (vLLM lists NIXL, LMCache, Mooncake and others). A 10K-token context is a lot of data, so you need a fast interconnect, and the transfer adds to time to first token. - You must size two pools and keep their ratio right as traffic changes, or one side idles while the other queues. - The vLLM feature is marked experimental.

Try the cheaper option first. Chunked prefill with a smaller token budget per step reduces the same interference; vLLM notes it can achieve the same goal but that the right chunk size is hard to find in practice. Benchmark both against your real length distribution and measure goodput at the p99 SLO. Adopt disaggregation if chunked prefill cannot meet the tail SLO without giving up too much time to first token.

Trade-offs

Disaggregation gives separate, predictable control of time to first token and inter-token latency, at the cost of KV transfer, two pools to size and an experimental feature; chunked prefill is simpler but harder to tune for both targets.

Common mistake

Adopting disaggregation to raise raw throughput, then finding that tokens per second did not improve and operations got harder.

Follow-up

How would you choose the initial ratio of prefill to decode GPUs, and what signal would tell you to change it?

Question 39HardAdapters

You serve 200 customer LoRA adapters on one base model with vLLM. Most traffic hits about 20 of them. Latency is fine in tests with one adapter but poor in production, and some new adapters fail to load. What do you check and configure?

#

Adapters were trained by different teams with ranks from 8 to 64. Customers upload new adapters during the day.

Show answer

Check the engine limits, which are easy to leave at their defaults.

- --max-loras is the maximum number of adapters in a single batch, and its default is 1. With 20 hot adapters, requests for different adapters cannot share a batch, so they wait for each other. Set it near the number of adapters that are active at the same time, and measure, because each slot reserves GPU memory. - --max-lora-rank defaults to 16, so rank-32 and rank-64 adapters will not load. The vLLM docs say to set it to the largest rank you use and not higher, because oversized values waste memory. - --max-cpu-loras sets how many adapters are kept in CPU memory (it must be at least max_loras), so a cold adapter is moved from CPU rather than loaded from storage. - With tensor parallelism, --fully-sharded-loras shards all of the LoRA computation instead of half; the docs say it is likely faster at high sequence length, high rank or large tensor-parallel size.

For uploads during the day, vLLM supports runtime loading (VLLM_ALLOW_RUNTIME_LORA_UPDATING and /v1/load_lora_adapter) and resolver plugins, but its docs warn that runtime updating has security risks and should not be used in production unless the environment is isolated and fully trusted. Prefer a controlled pipeline: validate each adapter (rank, target modules, base model version), store it in a trusted location, and roll it out through configuration.

If traffic is very uneven, route the hottest adapters to dedicated replicas, or merge an adapter that carries most of the traffic into its own full model.

Trade-offs

Higher max-loras and max-lora-rank allow more mixed batches and bigger adapters, but reserve GPU memory that would otherwise hold KV cache.

Common mistake

Benchmarking with a single adapter and leaving --max-loras at 1, so mixed-adapter production traffic is serialised.

Follow-up

How would you stop one customer's adapter traffic from slowing down every other customer on the same replica?

Question 40HardMultimodal

You deploy a vision-language model on vLLM for invoice extraction. It fits easily in testing, but in production time to first token is high, API server CPUs are pegged, and KV cache capacity is far lower than for the text-only model of the same size. What is going on and what do you change?

#

Each request has one scanned page image, often at high resolution. The model also supports video, which you do not use. Tensor parallel 4.

Show answer

Several separate costs, each with its own fix.

Memory reserved for inputs you never send. vLLM's --limit-mm-per-prompt defaults to 999 items per modality. The conserving-memory guide says to allow fewer items per prompt to reduce the footprint, and to disable unused modalities by setting their limit to zero, so no memory is allocated for videos. For one invoice page, set images to 1 and video to 0.

Huge image token counts. A high-resolution image becomes many visual tokens, which all go through prefill and take KV cache. For models that support it, mm_processor_kwargs such as max_pixels (Qwen2-VL series) shrink the processed input. Check extraction quality at the lower resolution with your eval set, since small print can suffer.

CPU-bound preprocessing. Image decoding and resizing happen in the API server. vLLM supports scaling out input processing with --api-server-count when CPU capacity is available; its guide notes that this disables the multi-modal IPC cache and suggests adjusting VLLM_MEDIA_LOADING_THREAD_COUNT.

Encoder parallelism. By default the vision encoder is sharded with tensor parallelism like the decoder, but encoders are small and the per-layer all-reduce is expensive. Setting mm_encoder_tp_mode="data" runs the encoder data-parallel; vLLM reports about 10% better throughput and TTFT at tensor parallel 8, with slightly more memory use.

Also, the processor cache (mm_processor_cache_gb, default 4 GiB) only helps if the same images repeat, which invoices rarely do.

Measure each change separately: TTFT, CPU per request, KV cache usage and extraction accuracy.

Trade-offs

Lower image resolution and tighter input limits free memory and cut prefill time, but can lose fine print; more API servers fix CPU bottlenecks at the cost of the IPC cache.

Common mistake

Sizing a vision-language deployment like a text model and ignoring the visual tokens and the default per-modality limits.

Follow-up

How would you serve both single-image invoices and multi-page PDFs without sizing every replica for the worst case?

Question 41MediumLong context

A customer needs 128K-token inputs on a model whose native context is 32K, and the model card says YaRN extends it. You add the rope scaling config to the production deployment. A week later, quality on ordinary short chats has dropped slightly. Why, and how should you deploy long context instead?

#

Qwen3-8B served on vLLM. Fewer than 2% of requests are longer than 32K tokens.

Show answer

The Qwen3 model card explains it. The model natively supports 32,768 tokens and its long-context quality was validated up to 131,072 tokens with YaRN, using a rope_scaling config of type yarn with factor 4.0 and original_max_position_embeddings 32768. But the card warns that the notable open-source frameworks implement static YaRN: the scaling factor stays the same regardless of input length, which can hurt performance on shorter texts. So every short chat now runs with positions scaled for 128K.

Deploy it as the card recommends: add rope_scaling only when long context is needed, and choose the smallest factor that covers your real lengths. The card's example: if typical length is 65,536 tokens, a factor of 2.0 is better.

In practice: - Keep the default pool without rope scaling for the 98% of short requests. - Run a separate long-context pool with YaRN, a matching --max-model-len, and capacity planned for very large KV caches per request (a single long request can use as much KV cache as hundreds of short ones). - Route by input token count at the gateway, with a clear error or a summarise-first path when a request exceeds the long pool's limit. - Evaluate both pools: short-chat regression tests on the long pool, and long-context retrieval and reasoning tests (not just needle-in-a-haystack) at the target lengths.

Trade-offs

One pool with YaRN is simpler to run but taxes every short request; two pools keep short-chat quality and let you size the long pool separately, at the cost of routing and lower utilisation.

Common mistake

Turning on context extension for all traffic because a model card says the model supports it.

Follow-up

How would you plan KV cache capacity for the long-context pool so that a few 128K requests do not starve each other?

Question 42HardAutoscaling

Design the autoscaling signal for a vLLM deployment on Kubernetes. GPU utilisation sits near 100% whether the service is healthy or overloaded, so it is useless. Which metrics would you scale on, how would you configure the autoscaler, and what would you watch for?

#

Interactive chat with a time-to-first-token SLO. Replicas take several minutes to become ready. Traffic doubles at 9 a.m. and drops at night.

Show answer

Use signals that measure queueing and KV pressure, which rise before users feel it.

- vllm:num_requests_waiting (gauge): requests waiting to be processed. vLLM also exposes vllm:num_requests_waiting_by_reason, split into capacity and deferred, which tells you whether the wait is from lack of room. - vllm:kv_cache_usage_perc (gauge, 1 means 100%): when it stays near 1, the server is preempting or refusing to admit work. - vllm:request_queue_time_seconds (histogram): the most direct link to the TTFT SLO, useful as an alert and a check on the autoscaler.

Configure the autoscaler around them: - Scale on per-replica average waiting requests, with a target found by load test: the queue length at which your TTFT SLO starts to break. Add KV cache usage as a second metric. The Kubernetes HPA computes a desired replica count for each metric and takes the largest. - Know the defaults. HPA scales up immediately, adding at most 100% of current replicas or 4 pods per 15 seconds, whichever is more; scale-down waits for a 300-second stabilisation window; changes within a 10% tolerance are ignored. Lengthen the scale-down window for GPU pools, because removing a replica you need again in ten minutes means another slow start. - If you use KEDA's Prometheus scaler, threshold is the target value and activationThreshold decides when to scale from zero. Scale to zero only for workloads that can accept a cold start. - Because replicas start slowly, keep headroom (a minimum replica count, or scheduled scale-up before 9 a.m.), and work on start-up time itself.

Watch for oscillation, for scale-up that is blocked by no free GPU nodes (pending pods), and for metrics that are averaged across replicas behind a load balancer that is not load-aware, which hides one overloaded replica.

Trade-offs

Queue-based scaling reacts to real demand and links to the SLO, but needs load-tested targets; scaling on KV usage catches long-context pressure but can keep replicas around for a few long sessions.

Common mistake

Scaling on GPU utilisation or CPU, which do not change between healthy and overloaded LLM servers.

Follow-up

How would you autoscale a pool that serves both interactive chat and batch jobs with different SLOs?

Question 43MediumKV cache

At peak, end-to-end latency jumps for long answers and the vLLM logs fill with "preempted by PreemptionMode.RECOMPUTE because there is not enough KV cache space". There are no OOM errors. What is happening, and what are your options?

#

vLLM V1 on a single GPU per replica. max_num_seqs was raised last week to improve throughput.

Show answer

vLLM admits requests and allocates KV cache blocks as tokens are generated. When the cache cannot hold all running requests, it preempts some of them. In vLLM V1 the default preemption mode is recompute: the preempted request loses its KV cache and its prompt and generated tokens are processed again when space is available. That is why there is no OOM, but long answers, which are the most likely to be preempted and the most expensive to recompute, get much slower. vLLM's tuning guide warns that preemption and recomputation can hurt end-to-end latency.

Confirm it with metrics: vllm:num_preemptions (counter) and vllm:request_num_preemptions (histogram of how often each request was preempted), alongside vllm:kv_cache_usage_perc near 1.

The tuning guide lists the options: - Give the KV cache more memory: increase gpu_memory_utilization (or leave it at its default if it was lowered). - Run fewer requests at once: decrease max_num_seqs or max_num_batched_tokens. Last week's increase is the likely cause. - Spread the model: higher tensor or pipeline parallelism leaves more memory per GPU for KV cache.

Also consider a quantized KV cache (after calibration and long-context evals), a lower max_model_len for this pool, or routing long-output requests to a pool sized for them. Then raise throughput the safe way: add replicas, and keep the concurrency setting below the point where preemptions start.

Trade-offs

Higher concurrency raises throughput until the KV cache runs out; beyond that, preemption and recompute make latency worse for exactly the longest requests.

Common mistake

Raising max_num_seqs to improve a throughput benchmark with short outputs, then treating preemption warnings in production as harmless.

Follow-up

How would you find the highest max_num_seqs that keeps preemptions near zero for your real length distribution?