supraj.dev

Inference Internals · Part 3 of 5

What is continuous batching?Seat the next table every round.

One GPU serves many people at once. The trick that makes it work is simple to say: don't wait for the slowest answer. Re-plan the batch after every single word.

Supraj Maripeddi · · 14 min read

Cover: two timelines of four GPU seats. In the top one, three seats sit idle while the longest answer finishes. In the bottom one, a new request takes each seat the moment it frees up.
Start hereRead Part 1 and Part 2, or just know that a model writes one word at a time and keeps notes for each.
Leave knowingWhy waiting for the batch wastes the GPU, how re-planning every step fixes it, and where prefill and preemption fit.
Keep in mindBig speedups here are from 2022 and 2023 papers, on their own test setups. vLLM's scheduler has changed since.

Part 2 ended with a full shelf of trays and a busy restaurant. PagedAttention made room for far more tables. But room is only half the story. Someone still has to decide who sits down, and when.

That someone is the scheduler, and this part is about how it decides. We stay in the same kitchen. The cook writes each reply one word at a time and files a card per word. The host seats tables and keeps the pegboard. Now watch the host's other job: running the dining room.

01 / The reason

Why serve many people in the same step at all?

Because the slow part of writing a word is fetching, not thinking. Part 1 showed that each decode step has to read the model's weights from memory, and that this phase "severely underutilizes GPU computation and becomes memory-bound". [2]

So you share the fetch. The PagedAttention paper puts it plainly: requests share the same weights, so "the overhead of moving weights is amortized across the requests in a batch". [2] The Orca paper, which proposed the idea in this article, states its design rule the same way: do "as much computation as possible per each round of model parameter read". [1]

In the kitchen, that's one trip to the recipe books per round, and a word for every seated table on the way back. The more tables seated, the more useful each trip.

In the kitchen
Each round, the cook walks to the recipe books once and writes the next word for every seated table.
In the model
Each step reads the weights once and produces one token for every request in the batch.

02 / The old way

What goes wrong when everyone starts and finishes together?

The obvious way to batch is the way a set-menu restaurant runs sittings. Seat four tables at 7 pm, serve them together, and seat nobody new until all four have finished.

That's how the serving systems Orca studied worked, such as NVIDIA's Triton in front of FasterTransformer. The paper describes it: the system schedules "at request granularity", and "the engine maintains a batch of requests fixed until all requests in the batch finish". [1] The paper calls this request-level scheduling. I'll call it static batching.

The trouble is that answers have wildly different lengths. One person asked for "yes or no". Another asked for an essay. The paper lists two costs: [1]

  • Finished people can't leave. A short answer is done, but it isn't sent back until the whole batch finishes. The engine even keeps doing work for those finished requests until then.
  • New people can't sit. A request that arrives mid-batch waits for every request in the current batch to finish.

Both come from one decision: the batch is planned once and never changed. The PagedAttention paper sums up the choice a naive batcher faces: make "earlier requests wait for later ones or delay the incoming requests until earlier ones finish". [2]

In the kitchen
A fixed sitting. A table that finished in five minutes keeps its seats for an hour, while the next party waits at the door.
In the model
Request-level scheduling. The batch is fixed until its longest request ends, so short requests can't return and new ones can't join.

03 / The idea

What changes if you re-plan the batch every round?

Everything. Orca's answer, which it called iteration-level scheduling, is to plan one step at a time instead of one batch at a time. An iteration is one run of the whole model, producing one token per request. The scheduler repeats three moves: [1]

  1. Pick the requests to run next.
  2. Run one iteration for them.
  3. Take the results back.

Because the scheduler gets control back after every step, a finished request can be sent to its user straight away. And a new request only has to wait for the step that's already running. [1] The PagedAttention paper describes the same mechanism in one line: "After each iteration, completed requests are removed from the batch, and new ones are added." [2] That's what continuous batching means.

Here's the difference on a toy dining room with four seats.

Figure 1 · Same eight requests, two ways to seat them

Fixed sittings

Nobody new sits down until all four seats are done.

Re-plan every round

A free seat is filled at the very next step.

  • Writing a word (requests A to H)
  • Done, but the seat is still held
  • Empty seat

With fixed sittings, all eight requests finish after 13 steps. Re-planning every round finishes them after 10, and no seat is ever held by a finished request.

Toy numbers. Each column is one step, and each request writes one word per step. A, B, C and D arrive at step 1; E at step 2, F at 3, G at 4 and H at 6. They need 6, 2, 3, 8, 3, 5, 2 and 4 words. A request's prompt is folded into its first step here. Chapter 6 shows how real servers handle prompts.

Look at request E. With fixed sittings it arrives at step 2 and waits until step 9, because D is still writing its eighth word. Re-planning every round, it sits down at step 3, the moment B leaves.

In the kitchen
After every round, the host looks at the room. A finished table leaves, and the next party in line sits down before the cook's next trip.
In the model
Iteration-level scheduling. The scheduler re-forms the batch at every step, removing finished requests and adding waiting ones.

04 / The hard part

How can requests at different points share one step?

This is the bit that's easy to skip, and it's why Orca needed a second trick. Once requests join and leave freely, one step holds requests at very different points. One is reading a fresh prompt. One is on word 5. Another is on word 300.

The usual way to batch on a GPU needs every request in the same shape. The Orca paper names three cases that break it: two new prompts of different lengths, two answers at different word positions, and one request reading its prompt while another writes its answer. The paper says this "significantly reduces the likelihood of batching in real-world workloads". [1]

Orca's fix is called selective batching. Most of the model's operations, like the multiplications with the weights, don't care which request a token belongs to. So Orca lines up every token in the step, from every request, and runs those parts as one big batch. Only attention, the look-back at each request's own cards, is split and done per request. [1]

Isn't splitting attention wasteful? Less than you'd think. Attention has no weights of its own, so batching it wouldn't save any weight fetching anyway. The paper measured that this choice has "only a small impact on efficiency". [1]

vLLM does the same kind of thing. At each step it joins all the step's input tokens into one sequence, the whole prompt for a new request and the latest word for the others, and its PagedAttention kernel reads each request's own cards through its block table. [2]

In the kitchen
The cook does the shared prep for every table in one go. Only the step of reading a table's own cards is done table by table.
In the model
Weight operations run on all the step's tokens together. Attention runs per request, over that request's own KV cache.

05 / The link to Part 2

Why does continuous batching need PagedAttention?

Because seats aren't the only limit. Every seated request needs its tray of cards, and the shelf is fixed. The PagedAttention paper says it directly: even with fine-grained batching, the number of requests in a batch "is still constrained by GPU memory capacity". [2]

Orca itself handled memory the old way. When it first scheduled a request, it reserved room for that request's maximum possible length, so it could never run out halfway. [1] That's the roped-off ten-seat table from Part 2.

The vLLM team tested exactly this. Orca's code wasn't public, so they built their own versions of it: one that magically knew each answer's real length (Oracle), one that reserved up to twice the real length (Pow2), and one that reserved the model's maximum, 2,048 tokens (Max). [2] On their ShareGPT workload, vLLM handled 1.7 to 2.7 times the request rate of the Oracle version, and 2.7 to 8 times that of the Max version, at similar latency. For one model, OPT-13B, vLLM ran 2.2 times as many requests at once as Oracle. [2]

Notice what that means. Even an Orca that knew the future served fewer people at once. The paper's reason: PagedAttention "can efficiently manage the memory usage and thus enable batching more requests than Orca". Across its tests, the paper sums it up as 2 to 4 times the throughput at the same latency, compared with Orca and FasterTransformer. [2] The paper calls the two ideas "complementary": Orca keeps the seats busy, and PagedAttention fits more trays on the shelf. It also notes that scheduling like Orca's "makes memory management more challenging", because requests come and go at every step. [2]

There's one exception worth knowing. On the largest model with short requests, there was so much spare memory that Orca could batch plenty of requests anyway, and vLLM's lead shrank. [2] Paging helps most when memory is the limit.

In the kitchen
A host who seats people every round is only as good as the shelf. Small trays on demand let the host actually seat them.
In the model
Iteration-level scheduling keeps the batch full. Paged KV blocks decide how big that batch can be.

06 / Prompts and words

Where do new prompts fit in?

Part 1 split a request into two phases. Prefill reads the whole prompt at once and writes all its cards in parallel. Decode writes one word per step. vLLM's docs call prefill compute-bound and decode memory-bound. [3]

With continuous batching, a new request's prefill lands in the same step as everyone else's decode. A huge prompt, a whole document say, could make that step slow, and every seated table would wait for its next word.

Today's vLLM answers with chunked prefill, on by default "whenever possible" in its V1 engine. Each step has a token budget, max_num_batched_tokens. The scheduler first gives every running decode its token. Then it fills what's left of the budget with waiting prompts, and if a prompt doesn't fit, it cuts the prompt into chunks across several steps. [3] The vLLM team describes the V1 scheduler as having no separate prefill and decode phases at all: each step is just a list of how many tokens each request gets. [6]

Figure 2 · Filling one step's token budget

Step 1 ABCDDDDD D's prompt: 5 of 12
All 3 steps
  1. Over three steps, A, B and C never missed a word, while D's long prompt was read in chunks around them and E's short one slipped into the last gap.
Toy budget of 8 tokens per step. The real one is max_num_batched_tokens; vLLM's docs give 2048 and more than 8192 as examples.

The budget is a real trade-off, and vLLM's docs spell it out. A smaller budget (their example is 2048) gives better inter-token latency, because fewer prefills slow the decodes down. A bigger budget gives a better time to first token, because more prompt tokens get through each step. [3]

In the kitchen
A banquet order arrives. The cook reads a few lines of it each round, after writing the next word for every seated table.
In the model
Decodes are scheduled first. Prefills fill the rest of the token budget, split into chunks when they don't fit.

07 / Under pressure

What happens when the shelf runs out mid-meal?

Here's the catch with seating people eagerly. Nobody knows how long an answer will be. The paper notes that output lengths "are not known a priori". [2] So the host may seat a table and find, twenty rounds later, that there's no free tray for its next card.

Part 2 introduced the answer: preemption. vLLM pauses a running request, frees all its blocks, and puts it back in the waiting line. The paper's rules are first-come-first-served: the earliest requests are served first, and the latest are preempted first. A request loses all of its blocks or none. [2]

In today's vLLM, the default is to recompute, not to copy cards out to CPU memory. [3] A preempted request is put back in the waiting queue and will "re-start its prefill phase" when it's scheduled again. [5] In the V1 scheduler's code, running requests are planned before waiting ones. When there's no room, the default policy picks the request at the back of the running list, the one most recently let in. And in a step where it had to preempt, it doesn't seat anyone new. [7]

You'll know it's happening. vLLM logs a warning that a request "is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space", and its docs warn this "can adversely affect end-to-end latency". [3]

In the kitchen
No free tray. The host clears the newest table's trays, sends them back to the line, and re-reads their order when space comes back.
In the model
The scheduler frees a request's KV blocks and requeues it. On its return, its KV cache is rebuilt from the prompt and the words written so far.

08 / The limits

What doesn't continuous batching fix?

It keeps seats busy. That's all it promises. Here's what it leaves alone.

What continuous batching changes, and what it doesn't.
Fixed Not fixed
Finished requests waiting for the slowest one Memory. The batch is still capped by KV cache space. [2] That's Part 2's job.
New requests waiting for a whole batch The batch-size trade-off. Bigger batches raise throughput but can raise latency, with diminishing returns. [1]
Work spent on requests that are already done Unknown answer lengths, and so preemption under heavy load (chapter 7).
Requests at different stages sharing a step The cost of each step. Every step still reads the weights; batching only shares that read. [2]

The second row matters most in practice. Orca found that in its tests, a bigger maximum batch raised throughput without hurting latency. But it also warned there's "no guarantee" of that for other hardware, models and workloads, and the maximum "must be set carefully". [1]

09 / Hands-on

Which knobs matter in vLLM?

Like PagedAttention, you don't switch continuous batching on. It's how vLLM's scheduler works. The vLLM team describes vLLM as "a full-fledged continuous batching engine". [6] What you tune is how big each step can get, and who goes first.

The scheduler settings, from vLLM's docs.
Setting What it does In the kitchen
--max-num-seqs "Maximum number of sequences to be processed in a single iteration." [4] How many seats
--max-num-batched-tokens "Maximum number of tokens that can be processed in a single iteration." The chunked-prefill budget. [4] Lines the cook handles per round
--scheduling-policy fcfs (the default) serves in order of arrival. priority serves by a priority you give each request, lower first. [4] Who the host seats first

Then watch the dining room. vLLM's metrics include vllm:num_requests_running and vllm:num_requests_waiting, the seated tables and the line at the door, plus vllm:kv_cache_usage_perc for the shelf, and time to first token and inter-token latency for how it feels to users. [5]

What I'd do
  1. Leave continuous batching and chunked prefill alone. They're on by default for good reasons.
  2. Tune --max-num-batched-tokens for what your users feel: smaller for smoother streaming, bigger for a faster first word. The docs suggest more than 8192 for throughput, especially for smaller models on large GPUs. [3]
  3. As in Part 2, treat steady preemption warnings as a signal. Give the KV cache more room, or lower --max-num-seqs or --max-num-batched-tokens so fewer requests compete. [3]

Keep these names

A small glossary

Scheduler
The part of a serving system that decides which requests run in each step.
Iteration (step)
One run of the whole model, producing one token for each request in the batch.
Static batching (request-level scheduling)
A batch that stays fixed until every request in it has finished.
Continuous batching (iteration-level scheduling)
Re-forming the batch at every step, so finished requests leave and new ones join.
Selective batching
Orca's trick: batch the weight operations across all tokens, run attention per request.
Prefill and decode
Reading the whole prompt at once, and then writing one token per step.
Chunked prefill
Splitting a long prompt across several steps so it fits the token budget.
Token budget
The most tokens one step may process. In vLLM, max_num_batched_tokens.
Preemption
Pausing a running request and freeing its KV blocks when memory runs out.
Throughput and latency
How much work the server finishes per second, and how long one user waits.

Check the concept

Can you explain it back?

1. With static batching, why does a short answer wait for a long one?

The batch is fixed until all its requests finish, so the short answer can't be returned and its seat can't be reused until the longest one is done.

2. With continuous batching, how long does a new request wait for a seat?

If a seat is free, only for the step that's already running. The scheduler re-forms the batch after every step.

3. Why can Orca skip batching attention without losing much?

Attention has no weights of its own, so batching it wouldn't share any weight fetch. The weight operations are still batched across all tokens.

4. Orca already re-planned every step. Why did vLLM still get 2 to 4 times its throughput?

Orca reserved memory for each request's maximum length. vLLM's paged blocks wasted far less, so many more requests fit in each batch.

5. A 20,000-token document arrives while 50 chats are streaming. What does chunked prefill do?

Each step, the 50 chats get their next token first. The document fills the rest of the token budget, a chunk at a time, until its prompt is read.

Continue the thread

The rest of the kitchen

Read further

Sources

All accessed 18 November 2026. vLLM docs pages are the "latest" version on that day.

  1. Yu et al., "Orca: A Distributed Serving System for Transformer-Based Generative Models", OSDI 2022
  2. Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023
  3. vLLM docs: Optimization and Tuning (preemption, chunked prefill)
  4. vLLM docs: Engine Arguments
  5. vLLM docs: Metrics (design)
  6. vLLM blog: "vLLM V1: A Major Upgrade to vLLM's Core Architecture" (27 January 2025)
  7. vLLM source: V1 scheduler (vllm/v1/core/sched/scheduler.py), commit e3036e8

Written with AI assistance. Every technical claim was checked against the original sources listed above.