Articles Inference engineering
How AI actually answers you
A plain-language masterclass on inference engineering: the craft of making an AI model reply fast, cheaply and reliably to millions of people. We will run it like a restaurant kitchen, then open up vLLM, the open-source tool that many companies use as their kitchen manager.
No AI background needed. If you can picture a busy kitchen on a Saturday night, you can follow every page of this. Every technical claim is checked against the original papers and the official vLLM documentation, listed at the end.
What a language model is, in one bite
Forget robots and brains for a minute. A large language model (an LLM, like the ones behind ChatGPT, Claude or Gemini) does one simple thing, over and over:
That "little piece" is called a token. A token is usually a word or part of a word. "Unbelievable" might become three tokens, like Un believ able, while "cat" is just one. Roughly speaking, 100 tokens is about 75 English words. The exact split depends on each model's own dictionary.
The model's "knowledge" is stored as billions of numbers called weights (also called parameters). When you hear "a 7B model" or "a 70B model", the B means billion: 7 billion or 70 billion of these numbers. Think of them as everything a chef learned in ten years of cooking school, squeezed into one enormous recipe book.
In the kitchen
A chef with a giant recipe book. Each order is cooked one bite at a time, and each bite depends on the bites before it.
In the machine
A model with billions of weights. Each answer is generated one token at a time, and each token depends on the tokens before it.
Keep this picture in your head. Everything in inference engineering comes from that one fact: answers are produced one token at a time, in order, and each token needs the whole recipe book.
Training is school. Inference is the job.
An AI model lives two very different lives.
Training is cooking school. You feed the model a huge amount of text for weeks or months, on thousands of chips, and it slowly adjusts its weights until its guesses get good. It is expensive, but it happens rarely, and only the model makers do it.
Inference is the job. It is every single time someone types a question and the model produces an answer. The word just means "using the trained model to produce an output". It happens millions of times a day, forever, for as long as the model is in use.
In the kitchen
Cooking school happens once. Running the restaurant happens every night, for years. That is where most of the daily effort and the electricity bill go.
In the machine
Training builds the weights. Inference uses them for every request. For a popular product, inference is the part that runs all day, every day.
So what is inference engineering?
Inference engineering is the job of running that restaurant well. The book Inference Engineering by Philip Kiely (Baseten) describes inference engineers as people who work "across the stack from CUDA to Kubernetes" in pursuit of "faster, less expensive, more reliable serving" of AI models. In plain words, they care about four things:
- Speed. How quickly does the answer start appearing, and how smoothly does it stream?
- Cost. How many answers can one expensive machine produce per hour?
- Reliability. Does it keep working when ten times more people show up?
- Quality. Did any speed trick quietly make the answers worse?
These goals fight each other. Most of this masterclass is about that fight, and about how a tool like vLLM wins more of it.
Reading the order, then cooking the answer
Every request goes through two phases. Learn these two words and you understand half of inference engineering.
Phase 1: Prefill. The model reads your whole message (your prompt). It can read all the prompt's tokens at the same time, in parallel, because they are all already known. At the end of prefill it produces the very first token of the answer.
Phase 2: Decode. Now the model writes the rest of the answer, one token per step. It cannot write token 5 before it knows token 4, so this part is strictly one after another. Each new token is added to the text and fed back in to produce the next one.
In the kitchen
The chef glances at the whole order ticket at once (prefill). Then the chef plates the dish spoon by spoon, and each spoon depends on what is already on the plate (decode).
In the machine
Prefill processes all prompt tokens in parallel. Decode generates one token per step, feeding each new token back in. Long prompts make prefill heavy; long answers make decode long.
Why it is slow: the warehouse walk
Models run on GPUs: chips with thousands of tiny calculators that are excellent at doing lots of multiplications at once. A GPU has two parts that matter here:
- The calculators (the "compute"). Extremely fast.
- The GPU's memory (often called HBM or VRAM), where the weights live. Big, but data must travel from it to the calculators.
Here is the surprise. To produce one single token during decode, the GPU has to read through essentially all of the model's weights. A 7-billion-parameter model stored at 16-bit precision (2 bytes per number) weighs about 14 GB. So every token means moving roughly 14 GB from memory to the calculators.
Databricks' engineering team puts it like this: at small batch sizes, decode speed "is dependent on how quickly we can load model parameters from GPU memory". In everyday words: the stove is fast, the walk to the warehouse is slow.
In the kitchen
The recipe book is so huge that it lives in a warehouse. For every single spoonful, the chef must walk through the entire warehouse. The stove is idle most of the time.
In the machine
Decode is memory-bandwidth bound: the time per token is mostly the time to stream the weights from GPU memory to the compute units.
The big idea: one walk, many plates
If the chef must walk the whole warehouse anyway, why carry ingredients for just one table? Grab ingredients for 30 tables in the same trip. Each trip takes a little longer, but you feed 30 tables instead of one.
That is batching: running many people's requests through the model in the same step, so the weights are read once and used many times. It is the single most important idea in inference engineering. Try it below.
The chef's notepad: the KV cache
There is a second problem. To pick the next token, the model has to "look back" at every token so far, using a step called attention. During attention, each token produces two small bundles of numbers, called a key and a value. You can think of them as "what this word is about" (key) and "what this word contributes" (value).
Without a memory, the chef would have the memory of a goldfish. To write word 100, it would recompute the keys and values for words 1 to 99 again from scratch. Then for word 101, all 100 again. The work explodes.
So the model keeps a notepad. After computing the key and value for a token, it writes them down and reuses them for every later step. That notepad is the KV cache (K for keys, V for values). Each new step only computes notes for the newest token, then reads the old notes.
The catch: the notepad is big, and it grows
The notes are stored for every token, at every layer of the model, and they sit in the same precious GPU memory as the weights. Real numbers:
- The vLLM team reported that the KV cache "takes up to 1.7GB for a single sequence in LLaMA-13B".
- Databricks calculated about 0.31 GiB per 1,024-token conversation for Llama-2-70B, so 64 such conversations need about 20 GiB.
And you never know in advance how long an answer will be. One person wants "yes or no", another wants a 3,000-word essay. So the notepad is the thing that limits how many customers you can seat at once. Batching wants many customers; the KV cache needs room for each of them. Managing that memory well is exactly where vLLM made its name.
In the kitchen
Each table gets a notepad that grows with every course. The counter space for notepads is limited, so the number of notepads decides how many tables you can serve at once.
In the machine
Each request has a KV cache that grows with every token. GPU memory left over after the weights decides how many requests can be in a batch.
How we keep score
Inference engineers measure a handful of numbers. You will see these names on every dashboard and in every benchmark.
| Metric | Plain meaning | Kitchen version |
|---|---|---|
| TTFT | Time to first token. How long until the answer starts appearing. | How long until the first bite reaches your table. |
| ITL / TPOT | Inter-token latency, or time per output token. The gap between words while streaming. 100 ms per token means 10 tokens per second for that person. | How long between bites once the meal has started. |
| Latency | Total time for the whole answer: TTFT + TPOT × (number of tokens after the first). | From ordering to the last bite. |
| Throughput | Tokens per second produced across all users together. | Meals per hour for the whole restaurant. |
| Cost per token | What it costs you to produce, say, a million tokens. Mostly GPU rental divided by throughput. | Cost of ingredients and staff per plate. |
Meet vLLM, the kitchen manager
You now know the three enemies: memory walks (decode is slow), notepad space (the KV cache fills memory) and uneven orders (some answers are short, some are long). An inference engine is the software that fights all three. It loads the model onto the GPUs, accepts requests, decides who gets cooked when, manages the notepads and streams answers back.
vLLM is one of the most widely used open-source inference engines. It describes itself as "a fast and easy-to-use library for LLM inference and serving" and its slogan is "Easy, fast, and cheap LLM serving for everyone".
A short origin story
vLLM came out of UC Berkeley in 2023, together with a research paper that introduced an idea called PagedAttention (Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023). One of its first big real-world deployments was LMSYS's Chatbot Arena, which was serving the Vicuna chatbot to millions of users. Their original backend became a bottleneck as traffic grew. After switching to vLLM, the team reported handling up to 5× more traffic and cutting the number of GPUs needed by 50%.
Today vLLM supports 200+ model architectures and runs on NVIDIA and AMD GPUs, several CPU types, and, through plugins, Google TPUs and other accelerators. It can act as a drop-in replacement for the OpenAI API, so apps written for OpenAI can talk to your own model with almost no code change.
In the kitchen
The kitchen manager does not cook. They seat customers, hand out notepad pages, decide whose dish goes on the stove next, and make sure no counter space is wasted.
In the machine
vLLM does not invent the model. It schedules requests, manages KV cache memory, batches work for the GPU, and streams results back through an API.
vLLM's tricks, one by one
Each trick below solves one of the problems from the earlier chapters. For each one you get the kitchen story, the real mechanism and a picture.
Trick 1 · PagedAttention: notepad pages instead of reserved tables
The old problem. Before vLLM, most systems reserved one big, unbroken chunk of memory for each request's notepad, sized for the longest possible answer. If you reserved space for 2,000 tokens and the answer was 50 tokens, the rest sat empty. Gaps between chunks also went unused. The vLLM team measured that existing systems "waste 60% – 80% of memory due to fragmentation and over-reservation".
vLLM's fix borrows an old idea from computer operating systems, called virtual memory with paging. The KV cache is cut into small, fixed-size blocks, each holding the notes for a fixed number of tokens (16 tokens by default for most models; some attention backends and model types use larger blocks). Blocks are handed out only when a request actually needs more room, and they do not have to sit next to each other. Each request keeps a small block table that says "my 1st page is in slot 7, my 2nd page is in slot 2", like a book's index.
Old way: reserve the maximum, in one piece
vLLM: hand out 4-slot pages on demand
Because less memory is wasted, more requests fit in a batch, and bigger batches mean more throughput (Chapter 4). That chain is why PagedAttention mattered so much. In their 2023 launch, the vLLM team reported up to 24× higher throughput than Hugging Face Transformers and up to 3.5× higher than Hugging Face's Text Generation Inference, on their test setups at the time. Those other tools have improved since, so treat these as historical numbers, not today's comparison.
Bonus: sharing pages. Because pages are small and labelled, several requests can point to the same page, the way three photocopies can point to one original. In the original 2023 vLLM, asking for three answers to one prompt shared the prompt's pages and copied a page only when one answer needed to change it (called copy-on-write); the team reported this cut memory use for such cases by up to 55%. Current vLLM (V1) gets a similar effect through prefix caching (Trick 3): identical full pages are shared, and each page counts how many requests are using it.
In the kitchen
Instead of reserving a whole long table for every party "in case 20 friends show up", the manager hands out notepad pages one at a time, from any free shelf, and keeps a list of which pages belong to whom.
In the machine
The KV cache is split into fixed-size blocks, allocated on demand and stored non-contiguously. A per-request block table maps logical blocks to physical blocks.
Trick 2 · Continuous batching: no waiting for the slowest eater
The old problem. With simple batching, a group of requests starts together and the group is only finished when the longest answer is done. Someone who asked "what is 2+2?" sits at the table, done, while the GPU slot they occupy does nothing, waiting for someone else's 900-word essay.
The fix. vLLM schedules work every single step. The moment one request finishes, a waiting request takes its seat in the very next step. This is called continuous batching (also "in-flight" or "iteration-level" batching). Databricks reports that continuous batching "can achieve 10x-20x better throughput than dynamic batching".
In the kitchen
A conveyor-belt sushi bar. When one guest leaves, the next person in line sits down right away. No one waits for the whole row to finish.
In the machine
The scheduler re-forms the batch at every iteration, adding new requests and removing finished ones, so GPU capacity is not left idle.
Trick 3 · Prefix caching: prep the shared base once
Many requests start with the exact same text. A company chatbot might put the same 2,000-token instruction ("You are a helpful assistant for Acme Bank. Never…") in front of every message. Or a user asks ten questions about the same long PDF. Without help, the model re-reads that same text every time.
With automatic prefix caching, vLLM keeps the notepad pages of text it has already read, labelled with a fingerprint (a hash). Each page's fingerprint is built from the page's own tokens plus the fingerprint of the page before it, so a match means "the whole text up to here is identical". A new request whose beginning matches simply reuses those pages and skips that part of prefill. Only completely full pages are cached, and when memory is needed, the least recently used unused pages are thrown out first. In current vLLM versions this is switched on by default.
In the kitchen
Every dish starts with the same base sauce, so the kitchen makes the base once and ladles it out for each order. Only the toppings are made fresh.
In the machine
KV cache blocks for an identical prompt prefix are found by hash and reused across requests, skipping their prefill computation.
Trick 4 · Chunked prefill: don't let one giant order block the line
Imagine someone pastes a 100-page contract and asks for a summary. Its prefill is a huge job. If the kitchen does it all in one go, everyone who is mid-answer stops streaming until it is done. Their words freeze.
vLLM's scheduler handles this with chunked prefill, which is on by default in vLLM V1. In each step, it first takes care of everyone who is mid-answer (decode gets priority). Then it fills the remaining room in that step with prompt-reading work, cutting a big prompt into chunks if it does not fit. The size of that room is a setting called max_num_batched_tokens. Smaller values keep streaming smooth (better ITL); larger values get first tokens out sooner (better TTFT).
In the kitchen
A party orders 300 dumplings. The chef folds a few dozen between serving the other tables, instead of making everyone else wait an hour.
In the machine
Long prefills are split into chunks and batched together with decode steps, balancing compute-heavy prefill with memory-heavy decode.
Trick 5 · Speculative decoding: a junior chef guesses, the head chef checks
Remember: decode is slow because of the warehouse walk, and each walk normally gives one token. What if one walk could confirm several tokens? A small, fast helper (a "draft") guesses the next few tokens. The big model then checks all those guesses in a single step. Every guess it agrees with is kept for free. At the first wrong guess, the big model's own choice replaces it and the rest are dropped.
The important part: the final text follows the big model's own choices, so quality does not drop. vLLM's documentation describes its speculative decoding as "theoretically lossless up to the precision limits of hardware numerics", and says it is aimed at "medium-to-low QPS (queries per second), memory-bound workloads". In plain words, it helps most when the kitchen is not already packed. vLLM supports several drafting methods, including small draft models, EAGLE, multi-token prediction (MTP) heads built into some models, and simple n-gram matching that copies likely continuations from the text itself.
In the kitchen
A junior chef pre-plates the next five spoonfuls. The head chef tastes them all in one go, keeps the good ones, and fixes the first bad one.
In the machine
A cheap proposer drafts k tokens; the target model verifies them in one forward pass and accepts the longest correct prefix plus one token of its own.
Trick 6 · Quantization: a thinner recipe book
The weights are just numbers. Normally each one takes 16 bits (2 bytes). Quantization stores them with fewer bits, for example 8 bits (FP8 or INT8) or 4 bits (INT4, AWQ, GPTQ). A 14 GB book becomes about 7 GB at 8 bits. A thinner book means a shorter warehouse walk per token and more free memory for notepads. vLLM's docs put the trade-off simply: quantization "trades off model precision for smaller memory footprint". Well-made 8-bit versions are usually very close to the original; very aggressive compression can noticeably hurt quality, so you always test on your own tasks. vLLM can also store the KV cache itself in 8 bits to fit more notepads.
Trick 7 · Parallelism: when one kitchen is not enough
A 70-billion-parameter model at 16 bits is about 140 GB. That is more than most GPUs hold. Only the very largest chips (192 GB and up) can fit it alone, and even then many teams split it across GPUs to leave more room for notepads and to go faster. vLLM supports four main ways, and you can combine them.
Every page split across chefs
Each layer's numbers are sliced across GPUs that work on the same step together. The most common choice inside one server. Lots of chatter between GPUs, so it needs fast links.
Each chef owns some chapters
Layers are divided between GPUs, and work passes from one to the next like an assembly line. Useful across several servers.
Open a second identical kitchen
The whole model is copied, and each copy serves different customers. It adds throughput, not the ability to run a bigger model.
Specialist chefs at different stations
For "mixture of experts" models, which route each token to a few specialist sub-networks. Different experts live on different GPUs.
And when the counter overflows: preemption
Sometimes many answers grow long at once and the notepad space runs out. vLLM then preempts a request: it pauses it, frees its pages, and later recomputes its notes when room comes back. Your answer still arrives, just later. If this happens often, vLLM logs a warning, and the fix is usually to give it more memory, run fewer requests at once, or spread the model over more GPUs.
Inside vLLM: who does what
When you start vLLM's server, it runs as a few cooperating programs (processes). This is how the current version (called V1) is organized, according to its architecture docs.
API server
The front door and waiter. Takes HTTP requests (OpenAI-style), turns text into tokens, streams results back.
Engine core
The kitchen manager. Runs a non-stop loop deciding what runs in the next step.
GPU workers
The cooks. One process per GPU loads its share of the weights and runs the model. (With a single GPU, the worker runs inside the engine core.)
vllm serve <model> -tp=4 on one server with 4 GPUs runs 1 API server + 1 engine core + 4 GPU workers = 6 processes.Follow one request through it. You send a message. The API server turns it into tokens and passes it to the engine core. The scheduler puts it in the queue, gives it notepad pages (reusing any cached prefix), and slots its prefill into the next step. Every step, the GPU workers run the model for the whole batch at once. Each new token goes back through the API server to your screen, until the answer is complete and its pages are freed for the next customer.
Try it yourself
You need a Linux machine with an NVIDIA GPU and Python 3.11 to 3.14 (other hardware is supported too; see the installation guide). These commands are taken from vLLM's official quickstart.
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=autovllm serve Qwen/Qwen2.5-1.5B-Instruct
# port 8000 by default; try http://localhost:8000curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"prompt": "San Francisco is a",
"max_tokens": 7,
"temperature": 0
}'from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
reply = client.chat.completions.create(
model="Qwen/Qwen2.5-1.5B-Instruct",
messages=[{"role": "user", "content": "Explain a KV cache in one sentence."}],
)
print(reply.choices[0].message.content)Safety note. By default the server listens on all network interfaces and does not check API keys, so other machines on your network can reach it. To keep it private, add --host 127.0.0.1. Before exposing it, add --api-key (or set VLLM_API_KEY), and know that this protects only the main API routes such as /v1; also put the server behind a firewall or reverse proxy.
The knobs you will actually touch
| Setting | What it controls | Kitchen version |
|---|---|---|
| --gpu-memory-utilization | Share of total GPU memory this vLLM instance may use (weights, working memory and KV cache). Current source default: 0.92 (92%). | How much of the counter the manager may claim. |
| --max-model-len | Longest conversation (prompt + answer) allowed, in tokens. Lower it to save memory. | Biggest order the kitchen accepts. |
| --max-num-seqs | Most requests allowed in one batch. | Most tables served at once. |
| --max-num-batched-tokens | Token budget per step. Smaller favors smooth streaming (ITL); larger favors fast first tokens (TTFT) and throughput. | How much work fits on the stove per round. |
| --tensor-parallel-size | How many GPUs share each layer. | How many chefs share each page. |
| --kv-cache-dtype | Store the KV cache in a smaller format (for example fp8) to fit more requests. | Smaller handwriting on the notepads. |
| --speculative-config | Turn on speculative decoding and choose the method. | Hire the junior chef. |
Defaults change between releases, and several of them depend on your GPU. Check vllm serve --help for the version you install. This page was checked against vLLM's documentation and source code in October 2026.
Cheat sheet and quick quiz
- Token
- A small piece of text, often a word or part of one. Models read and write in tokens.
- Weights / parameters
- The billions of numbers that hold what a model learned. The recipe book.
- Inference
- Using a trained model to produce an answer. Running the restaurant.
- GPU memory (HBM / VRAM)
- Fast memory on the GPU that holds weights and the KV cache. The warehouse.
- Prefill
- Reading the whole prompt in parallel and producing the first token. Compute-heavy.
- Decode
- Writing the answer one token per step. Limited mostly by memory speed.
- KV cache
- Saved keys and values for past tokens, so they are not recomputed. The notepad.
- Batching
- Processing many requests in the same step so the weights are read once for all of them.
- TTFT
- Time to first token: wait until the answer starts.
- ITL / TPOT
- Time between tokens while the answer streams.
- Throughput
- Total tokens per second across all users.
- PagedAttention
- vLLM's method of storing the KV cache in small fixed-size blocks, allocated on demand.
- Continuous batching
- Re-forming the batch every step so finished requests are replaced right away.
- Prefix caching
- Reusing KV cache blocks for identical starting text across requests.
- Chunked prefill
- Splitting long prompts into pieces and mixing them with decode work.
- Speculative decoding
- A cheap helper drafts tokens; the big model verifies several at once.
- Quantization
- Storing numbers with fewer bits to save memory and speed up reading.
- Preemption
- Pausing a request and recomputing it later when KV cache memory runs out.
Check yourself
1. Why is writing an answer slower than reading the question?
Reading (prefill) handles all prompt tokens in parallel. Writing (decode) must produce one token at a time, and each step has to stream the model's weights from GPU memory, so it is limited by memory speed.
2. Why does batching help so much?
The weights are read once per step no matter how many requests are in the batch. Serving 32 requests in that step costs only a bit more than serving one, so total throughput rises a lot, while each person's stream slows only a little.
3. What problem did PagedAttention solve?
Wasted KV cache memory. Older systems reserved a large contiguous chunk per request for the maximum length, wasting 60–80% of the memory set aside for the KV cache. vLLM allocates small blocks on demand, so waste drops below 4% and many more requests fit in a batch.
4. Your chatbot sends the same long instructions with every message. Which vLLM feature helps?
Automatic prefix caching. The instructions' KV cache blocks are reused, so prefill for that part is skipped and the first token arrives sooner.
5. Does speculative decoding make answers worse?
No, when implemented correctly. The big model verifies every token, and vLLM describes its implementation as lossless up to hardware numerical precision. It mainly helps when the server is not already fully loaded.
Next in this masterclass series
Sources
- vLLM documentation, home and feature list
- vLLM docs source on GitHub: quickstart, optimization and tuning (preemption, chunked prefill, parallelism), architecture overview, automatic prefix caching, speculative decoding, quantization
- vLLM design doc: prefix caching
- vLLM blog: "vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention" (June 2023)
- Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023
- Databricks: LLM Inference Performance Engineering: Best Practices
- Philip Kiely, Inference Engineering (Baseten)