supraj.dev

Inference Internals · Part 4 of 5

What is speculative decoding?Let the apprentice guess ahead.

A big model writes one word per trip to its weights. Speculative decoding lets a small model guess the next few words, then has the big model check them all in one trip. The output follows the same odds as the big model alone. It just arrives sooner.

Supraj Maripeddi · · 15 min read

Cover: a small draft model proposes five words. The big model checks all five in one pass, keeps the first three, rejects the fourth and writes its own word in its place.
Start hereRead Part 1 to Part 3, or just know that a model writes one token per step and each step reads all its weights.
Leave knowingWhy checking guesses is cheap, the rule that keeps the output identical, and when speculation helps or hurts.
Keep in mindSpeedups quoted here come from each paper's own models and hardware, mostly at batch size 1. Yours will differ.

Part 3 ended with a busy dining room. The host fills every free seat at every round, and the cook writes one word for each table per trip to the recipe books. That keeps the kitchen full. It doesn't make any single table's reply arrive faster.

This part is about that one table. We stay in the same kitchen: the cook is the big model, the recipe books in the glass room are its weights, and the tray of cards is the KV cache. Now we hire an apprentice, a small, quick cook who guesses what the head cook is about to write.

01 / The problem

Why does every word cost a whole trip to the recipe books?

Because the model can't write word 11 until it has written word 10. Leviathan and colleagues at Google open their paper with exactly this: "decoding K tokens takes K serial runs of the model". [1]

Each of those runs is slow for a boring reason. It isn't the maths. It's the fetching. Chen and colleagues at DeepMind write that sampling is "typically memory bandwidth bound", so the time for one token is roughly proportional to the size of the weights and of the model's memory, its KV cache. [2] The Medusa paper says the same thing plainly: each step moves "the full model parameters" from the GPU's main memory into its cache, to produce a single token. [4]

So the chip spends most of a step waiting for data, with spare arithmetic sitting idle. Leviathan's paper builds on that observation: inference is often limited by "memory bandwidth and communication, so additional computation resources might be available". [1] Speculative decoding is a way to spend that spare arithmetic.

In the kitchen
The cook walks to the glass room, reads the books, and walks back with one word. The walk is the slow part. The writing takes no time.
In the model
Each decode step reads all the weights from memory to produce one token. The GPU's compute is mostly idle while it waits.

02 / The idea

What if a quick apprentice guessed the next few words?

That's the whole trick. Two teams published it independently: Leviathan et al. called it speculative decoding, and Chen et al. called it speculative sampling. Chen's paper says the "core underlying idea is the same". [2] One round works like this: [1] [2]

  1. Draft. A small, fast model writes a short guess, say 4 tokens, one after another. This is the draft model. The big one is the target model.
  2. Verify. The target model reads all 4 guesses in one run, and works out what it would have written at each position.
  3. Accept. Going left to right, keep guesses until the first one that fails a check. At that spot, the target writes its own token instead. If every guess passes, the target adds one extra token for free.

The name comes from processors. Speculative execution means doing a task in parallel with checking whether it was needed, like branch prediction. [1] Here, the "task" is the target model's work on words that might not survive.

Notice the safety net. Every round produces at least one token from the target model, so the target never runs more times than plain decoding would, even in the worst case. At best, one run produces the whole draft plus one: 5 tokens for a 4-token guess. [1]

In the kitchen
The apprentice scribbles the next four words on a slip. The head cook makes one trip, ticks the good ones, and writes the right word where the apprentice first went wrong.
In the model
A draft model proposes k tokens. The target model scores all of them in one forward pass, accepts a prefix, and samples one more token of its own.

03 / The cheap part

Why is checking four guesses as cheap as writing one word?

Because the target model already reads in parallel. Part 1 showed prefill: the model reads a whole prompt in one run. Checking a draft is the same move on a few tokens. Chen et al. state the key observation directly: scoring a short continuation in parallel has "a very similar latency" to sampling a single token. [2]

They give three reasons, for large models at small batch sizes: [2]

  • The weight multiplications stay memory-bound. With only a few tokens per step, reading the weights still dominates, so 5 tokens take about as long as 1.
  • The KV cache read doesn't grow. The cards for the earlier words are read once either way.
  • Communication between chips stays latency-bound. A few more tokens' worth of data barely changes the time of each exchange.

Leviathan's paper adds the memory view: the target's weights and KV cache are read once per round, so the number of reads shrinks by the same factor as the number of runs. [1] Here's a round in the kitchen, step by step.

Figure 1 · Three rounds of draft, verify, accept

So far

  • Apprentice's guess
  • Kept by the cook
  • Rejected
  • Thrown away after a rejection
  • Written by the cook

Three rounds wrote 10 words with 3 trips by the cook. Writing alone, the cook would have made 10 trips.

Toy example with a draft of 4 words. Which guesses pass is made up for teaching. In a real run it depends on the rule in chapter 4, and the apprentice's own time is not counted here.
In the kitchen
Reading four slips on one trip costs the cook about the same walk as reading one. The walk was always the expensive part.
In the model
Verifying k draft tokens is one forward pass over k + 1 positions. At small batch it costs close to one ordinary decode step.

04 / The guarantee

How can you keep a guess without changing the answer?

This is the clever part, and the reason speculative decoding is safe to switch on. The output isn't "close to" what the big model would write. It comes from the same probability distribution. [1] [2]

For each draft token, both models have a probability for it. The rule, a form of rejection sampling, goes like this: [1] [2]

  1. If the target likes the token at least as much as the draft did, keep it.
  2. If the target likes it less, keep it only some of the time: with a chance equal to the target's probability divided by the draft's.
  3. If it's rejected, throw away it and every guess after it. Then sample a replacement from what's left over: the tokens the target wanted more than the draft offered.

A tiny example. The draft model was 60% sure the next word is "dal". The target model is only 30% sure. So "dal" is kept half the time (30 ÷ 60). When it's thrown out, the replacement comes from words the target rated higher than the draft did. Leviathan's paper proves that for any pair of models, a token chosen this way follows the target's distribution exactly. [1]

Settings like temperature, top-k and nucleus sampling fit in too. Both papers reshape the probabilities first, then apply the rule. [1] [2] Chen's paper is careful about one thing: the match holds "within hardware numerics". Their speculative and normal runs couldn't be identical word for word, because of different random seeds and floating-point paths, so they checked that benchmark scores matched instead. [2] vLLM's docs give the same caveat: lossless "up to the precision limits of hardware numerics". [5]

In the kitchen
The cook keeps a guessed word with a fair coin, weighted by how much more the apprentice liked it. Diners can't tell who wrote which word.
In the model
Accept with probability min(1, target ÷ draft). On rejection, resample from the normalised leftover, max(0, target − draft).

05 / The key number

What decides how many guesses survive?

How often the draft agrees with the target. Leviathan's paper calls it the acceptance rate, written α, and shows it equals the overlap between the two models' distributions. [1] If they always agree, α is 1. If they never do, it's 0.

What moves it, according to the papers:

  • The draft model's size. In Leviathan's tests, α rose with the draft's size. Drafts "a couple of orders of magnitude smaller" than the target "tend to produce α values between 0.5 and 0.9". [1]
  • The sampling settings. Sharper distributions gave higher α. In their T5 tests, greedy decoding (temperature 0) gave higher α and bigger speedups than temperature 1. [1]
  • The task. Chen et al. found acceptance "dependent on the application". Code sped up more than summaries. Their guess: code is full of predictable patterns, like for i in range(len(arr)):. On Chinchilla 70B with a 4-token draft at batch size 1, they measured 1.92× on summaries and 2.46× on code, both with nucleus sampling. [2]

The paper also gives a formula. If each guess passes with chance α, a round with a k-token draft produces on average (1 − αk+1) ÷ (1 − α) tokens. Then you pay for the draft's own time: c, its run time as a fraction of the target's. [1] Try it.

Figure 2 · Acceptance rate, draft length and speedup

Tokens per target pass3.36

Estimated speedup2.80×

Estimated speedup for each draft length, best in green

At α 0.80 and c 0.05, a 4-token draft gives about 3.36 tokens per target pass and an estimated 2.80× speedup. The best draft length here is 8.

Leviathan et al.'s Theorem 3.8 and Equation 1. It assumes each guess passes independently with chance α and that the GPU has enough spare compute to verify for free. Real systems fall short of both.

The paper's own table matches this. With α 0.8, a 5-token draft and c set to 0, it gives 3.69× speed. [1] Its real measurements on T5-XXL (11B) came close: with T5-small as the draft, greedy translation ran 3.4× faster at α 0.75. [1]

In the kitchen
An apprentice who knows the house style gets more words ticked. An apprentice who writes slowly eats into the time saved.
In the model
Speedup depends on α, how often draft tokens are accepted, and c, how long the draft takes relative to the target.

06 / The dial

How many words should the apprentice guess?

More isn't better. A guess only counts if every guess before it passed, so the tenth word of a draft rarely survives. Chen et al. measured this on Chinchilla 70B: the share of accepted tokens falls as the draft gets longer, while each round's time grows roughly in line with the draft length. Past a point the speedup "plateaus or even regresses". For their summarisation task, latency was lowest at a 3-token draft. [2]

They found a second cost. Longer drafts made the time for a full answer more variable, which matters if you care about your slowest requests, the P90 and P99. [2] Leviathan's paper picks the best length numerically from α and c (its Figure 3), which is what the bars in Figure 2 do. [1]

In the kitchen
Guessing ten words ahead mostly wastes ink. Three or four good guesses beat ten hopeful ones.
In the model
Later draft tokens need all earlier ones accepted. The best k grows with α and shrinks as draft cost c grows.

07 / The variants

Who should do the guessing?

The acceptance rule works with any draft at all. Leviathan's paper says the guarantee holds "for any choice" of draft model. [1] So people have tried many kinds of apprentice.

Kinds of draft, as their sources describe them.
Draft How it guesses In the kitchen
Separate small model A smaller model of the same family. Chen et al. trained a 4B draft for Chinchilla 70B: 1.8 ms per token against 14.1 ms. [2] A junior cook trained in the same kitchen
N-gram, prompt lookup Copy a phrase that already appeared in the prompt. Almost free. Leviathan's paper suggested copying from context for summaries and chats where text repeats. [1] [13] Flipping back through the order slip
EAGLE A small head, one decoder layer, that predicts the target's own next internal features and reuses its embedding and output layers. [3] An apprentice reading the cook's rough notes
Medusa Extra output heads on the target's last hidden state, each guessing a word further ahead, checked together as a tree. [4] Extra hands on the cook, each jotting a later word
MTP The target model has multi-token prediction built in, so no separate draft is needed. Only for model families that support it. [7] A cook trained to jot the next word too

The trade-off is cost against accuracy. A bigram model is nearly free but weak: on translation it reached α of about 0.2, worth 1.25×. [1] EAGLE reports draft accuracy around 0.8 against about 0.6 for Medusa, and 2.7× to 3.5× latency speedups on LLaMA2-Chat 70B. [3] Its heads are small: 0.24B trainable parameters for a 7B model. [3]

One caution on Medusa. Its Medusa-1 version trains heads on a frozen model and keeps the output lossless. Its optional "typical acceptance" scheme swaps the exact rule for a looser one that aims at similar quality, not the identical distribution. [4] vLLM's docs flag a lossy verifier of their own, FLy, as "not distribution preserving". [5] If exactness matters to you, check which rule you're running.

08 / The limits

When does it help, and when does it hurt?

Everything so far leaned on one assumption: spare compute. Leviathan's paper is blunt: the method "is not helpful for configurations where additional computation resources are not available". [1] It also does more total arithmetic, because rejected guesses are wasted work. [1]

Now recall Part 3. Continuous batching exists to use up that same spare compute, by sharing each weight read across many requests. The two ideas compete for it. EAGLE's authors put it this way: as batch size rises, "the available computational capacity of the GPU decreases", and the speedup shrinks. On Vicuna 7B, their speedup went from 2.90× at batch size 1 to 2.65× at 3 and 2.76× at 4. [3]

vLLM's docs explain the large-batch case. With speculation, each request sends k tokens to verification, so the effective batch becomes batch size times k. Past a critical size, speculation "negatively impacts" the time per output token. [8] That's why vLLM's page pitches it for "medium-to-low QPS" and "memory-bound workloads". [5]

Where speculative decoding tends to pay off.
Likely to help Likely to disappoint
Few requests at once, latency matters most Big batches, where the GPU is already busy. [3] [8]
Predictable text: code, edits, text copied from the prompt Low acceptance, so most guesses are wasted. [1]
A draft that is fast compared with the target A draft so slow that c eats the gain. EAGLE notes a 7B draft for a 13B target ran slower than plain decoding. [3]

09 / The link to Parts 2 and 3

How does it fit with batching and the KV cache?

In vLLM, speculation lives inside the same scheduler from Part 3. The vLLM team wrote that V1's {request_id: num_tokens} plan is "general enough" to cover speculative decoding. [12] In the scheduler code, a request's token count includes its draft tokens, so a request with 3 guesses asks for 4 slots in the step, not 1. [11]

The cards need room too. The scheduler reserves KV slots for those extra tokens. After verification, it subtracts the rejected ones from the request's count of computed tokens, so the request's progress rolls back to the last accepted token. [11] A separate draft model also gets its own KV cache, which vLLM allocates alongside the target's. [5]

That extra memory has a cost Part 2 would recognise: fewer tables fit. EAGLE's paper says it needs "slightly more CUDA memory". On one 24 GB RTX 3090 with Vicuna 7B, the largest batch fell from 8 to 7. [3] Even so, it reported about 2× throughput at those maximum batch sizes. [3]

In the kitchen
The host now books a few extra card slots per table for the apprentice's guesses, and frees the slots of the ones the cook crossed out.
In the model
Draft tokens are scheduled and get KV slots like any other token. Rejected ones are rolled back. A draft model adds its own KV cache.

10 / Hands-on

How do you turn it on and measure it in vLLM?

Unlike PagedAttention and continuous batching, this is something you configure. You pass one JSON object, --speculative-config. The docs' draft-model example pairs Qwen3-8B with Qwen3-0.6B, guessing 5 tokens at a time: [5] [6]

vllm serve <target-model> \
  --speculative-config '{
    "method": "draft_model",
    "model": "<draft-model>",
    "num_speculative_tokens": 5
  }'
Common keys, from vLLM's speculative decoding docs.
Key What it does
method Which kind of draft. Common values include draft_model, ngram, suffix, mtp, eagle3 and dflash. [5]
model The draft model or EAGLE head. Often not needed for n-gram or MTP. [5]
num_speculative_tokens The draft length k: tokens proposed per step. [5]
prompt_lookup_min / max For n-gram: the window size of the phrase to match in the prompt. Both 5 if you set neither. [5]

Then measure. vLLM's /metrics endpoint counts verification steps, draft tokens and accepted tokens: vllm:spec_decode_num_drafts_total, vllm:spec_decode_num_draft_tokens_total and vllm:spec_decode_num_accepted_tokens_total. Divide accepted by draft tokens for the acceptance rate. [9] Start the server with --per-request-spec-decode-metrics summary and each response also carries a mean_acceptance_length: tokens per verification step, from 1.0 (nothing accepted) to k + 1. [9]

One trap in the latency numbers. Inter-token latency and time per output token can disagree when one output bundles several tokens, and vLLM's metrics docs name speculative decoding as the example. [10] Compare like with like.

What I'd do
  1. Turn it on only for latency-bound serving with few requests at once. For throughput-bound batch jobs, measure before assuming it helps.
  2. Try the cheapest draft first. If prompts repeat text, n-gram costs no extra model. If your model ships MTP heads, the docs suggest starting at num_speculative_tokens 1. [7]
  3. Watch acceptance rate and mean acceptance length on real traffic, next to time to first token and time per output token.
  4. If load swings, look at dynamic speculation, which lowers k as concurrency rises, down to 0. [8]

Keep these names

A small glossary

Target model
The big model whose answers you want. It has the final say on every token.
Draft model (drafter, speculator)
Anything that cheaply proposes the next few tokens: a small model, a lookup or a trained head.
Draft length (k, γ, num_speculative_tokens)
How many tokens the draft proposes per round.
Verification
One target forward pass that scores every draft token at once.
Rejection sampling (the acceptance rule)
Keep a draft token with chance min(1, target ÷ draft); otherwise resample from the leftover.
Bonus token
The extra token the target adds when every draft token is accepted.
Acceptance rate (α)
How often a draft token is accepted.
Mean acceptance length
Average tokens produced per verification step, counting the target's own token.
Lossless
Same output distribution as the target alone, up to floating-point differences.

Check the concept

Can you explain it back?

1. Why can the big model check 4 guesses in about the time of 1 word?

Decode is limited by reading the weights, not by maths. Scoring a few tokens in one pass reads the weights once, so at small batch it costs about the same.

2. The draft model is terrible. Does the answer get worse?

No. The acceptance rule keeps the target's distribution for any draft. A bad draft just gets rejected more, so you lose speed, not quality.

3. Draft is 70% sure of a word, target is 35% sure. What happens?

It's kept half the time (35 ÷ 70). If rejected, the target samples a replacement from words it rated higher than the draft did, and the rest of the draft is thrown away.

4. Why not always guess 10 tokens ahead?

A token only counts if all earlier ones passed, so late guesses rarely survive, while each extra guess costs draft time. Chen et al. also saw more variable latency.

5. Why can speculation slow down a server running 300 requests at once?

Continuous batching has already used the spare compute. Verifying k tokens per request multiplies the work per step, so steps get slower and the gain turns into a loss.

Continue the thread

The rest of the kitchen

Read further

Sources

All accessed 18 November 2026. vLLM docs pages are the "latest" version on that day.

  1. Leviathan, Kalman and Matias, "Fast Inference from Transformers via Speculative Decoding", ICML 2023 (arXiv 2211.17192)
  2. Chen et al., "Accelerating Large Language Model Decoding with Speculative Sampling", DeepMind, 2023 (arXiv 2302.01318)
  3. Li et al., "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty" (arXiv 2401.15077)
  4. Cai et al., "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads", ICML 2024 (arXiv 2401.10774)
  5. vLLM docs: Speculative Decoding
  6. vLLM docs: Draft Models
  7. vLLM docs: MTP (Multi-Token Prediction)
  8. vLLM docs: Dynamic Speculative Decoding
  9. vLLM docs: Per-Request Acceptance Metrics
  10. vLLM docs: Metrics (design)
  11. vLLM source: V1 scheduler (vllm/v1/core/sched/scheduler.py), commit 81a8cb0
  12. vLLM blog: "vLLM V1: A Major Upgrade to vLLM's Core Architecture" (27 January 2025)
  13. vLLM docs: N-Gram Speculation

Written with AI assistance. Every technical claim was checked against the original sources listed above.