Inference Internals · Part 5 of 5
What is quantization?Write the recipe books in shorthand.
A model is billions of numbers, and every number takes memory. Quantization stores them with fewer bits. The model shrinks, decoding often gets faster, and you pay for it in accuracy. This part is about how big that bill is, and how to keep it small.
Part 1 ended with a tip: write the cards in FP8 and they take half the space. That tip has a bigger sibling. The cards are only part of what sits in GPU memory. The biggest single thing is usually the recipe books, the model's weights.
Parts 2 and 3 fitted more trays on the shelf and kept every seat busy. This part makes the books thinner. Same kitchen. The cook still writes one word per step and still carries the recipe books out of the glass room every time. The question is what happens when those books are rewritten in shorthand.
01 / The numbers
What numbers does a model actually store?
Each weight is a number, and each number is stored in a format: a fixed number of bits with a rule for what those bits mean. More bits means more possible values, and more memory. Here are the formats you'll meet.
| Format | Bits | What it is | 8B weights |
|---|---|---|---|
| FP32 | 32 | IEEE single precision. A common training format. [5] | 32 GB |
| BF16 | 16 | FP32 with fewer precision bits. vLLM describes it as "a balance between precision and range". [5] [14] | 16 GB |
| FP16 | 16 | IEEE half precision. [5] | 16 GB |
| FP8 E4M3 | 8 | 4 exponent bits, 3 mantissa bits. Largest value 448. [5] | 8 GB |
| FP8 E5M2 | 8 | 5 exponent bits, 2 mantissa bits. Largest value 57,344. [5] | 8 GB |
| INT8 | 8 | Evenly spaced whole numbers, used with a scale, often from −127 to 127. [4] | 8 GB |
| INT4 | 4 | Only 16 possible values, so it always needs a scale. | 4 GB |
The two families behave differently. A floating-point format (FP32, BF16, FP16, FP8) splits its bits into an exponent and a mantissa. Its steps are tiny near zero and wide far from it, a "non-linear sampling of the real numbers", as the FP8 paper puts it. An integer format (INT8, INT4) has evenly spaced steps, so it only works once you choose a scale that maps your numbers onto those steps. [5]
The two FP8 flavours trade one for the other. NVIDIA's docs say of E5M2: "The tradeoff of the increased dynamic range is lower precision of the stored values." [6] The FP8 paper, written by NVIDIA, Arm and Intel, recommends E4M3 for weights and activations and E5M2 for gradients, which only matter in training. [5]
Quantization is storing numbers in one of the smaller formats and keeping enough information, usually a scale, to map them back. SmoothQuant's one-line definition: it "maps a high-precision value into discrete levels". [3]
- In the kitchen
- The recipe books say "137.4 g of flour". Shorthand says "about 140". Fewer digits, thinner books, and bread that comes out very slightly different.
- In the model
- Weights trained in 16 or 32 bits are stored in 8 or 4 bits plus a scale, and expanded back when they're used.
02 / The prize
How much memory does each format save?
The weights are easy: number of parameters times bytes per number. The KV cache from Part 1 sits on top, and it has its own format setting. Whatever the weights stop using is room for more trays, and so more people at once.
The papers show how big this gets. SmoothQuant notes that GPT-3's 175 billion parameters "consume at least 350GB of memory" in FP16. [3] GPTQ squeezed OPT-175B to 3 bits, about 63 GB including the layers it kept in FP16, and ran it on one 80 GB A100. In FP16 the same model needed five of those GPUs. [1] Try the trade on one GPU yourself.
Figure 1 · Weights and cards on one GPU
Model
Weight format
KV cache format
GPU memory
- Weights
- KV cache for the chats that fit
- Left over
- Weights
- 16 GB
- One 8,192-token chat
- 1.07 GB
- Chats that fit
- 59
Llama 3 8B in BF16 takes 16 GB of weights. That leaves 64 GB, room for 59 chats of 8,192 tokens each with a BF16 KV cache.
Two things jump out. Moving the 8B model from BF16 to INT4 frees 12 GB, room for eleven more chats of that length. The 70B model doesn't fit on one 80 GB GPU in BF16 at all. In 8 bits it fits with room for only 3 chats. At 4 bits, 16.
- In the kitchen
- Thinner recipe books leave more shelf for trays. More trays, more tables seated at once.
- In the model
- Weight memory scales with bytes per parameter. What's freed becomes KV cache, which sets how many requests fit in a batch.
03 / The speed
Why does decoding get faster when the weights shrink?
Because decode waits on memory, not maths. Part 1 borrowed the Masterclass's name for it, the warehouse walk: every step carries all the weights to the cook to write one word. The AWQ paper measured this on an RTX 4090, which has 165 TFLOPS of compute and 1 TB/s of memory bandwidth. Any work doing fewer than 165 operations per byte fetched is limited by memory there, and decoding in FP16 did about 1. [2]
The paper's conclusion follows: since the maths is fixed, "the only way to improve the peak performance is to reduce the total amount of memory traffic". Weights are most of that traffic, so 4-bit weights raise the ceiling about four times. [2]
GPTQ showed it end to end. A 3-bit OPT-175B on one A100 wrote each token in 71 ms, against 230 ms for FP16 spread across five A100s, about 3.25 times faster. On cheaper A6000 GPUs the gap was 4.5 times. Its kernel expands the weights back on the fly, and it "does not require any activation quantization". [1]
Two catches. First, the speed comes from kernels, not from the format alone. GPTQ notes that LLM.int8() "reduces memory costs but has the same runtime as the FP16 baseline". [1] LLM.int8()'s own paper says its overhead "can slow inference for models with less than 6.7B parameters". [4] Second, both GPTQ and AWQ measured at batch size 1, one person at a time. [1] [2] vLLM's docs pitch 4-bit weight-only models for "low queries per second (QPS)". [11]
- In the kitchen
- The cook carries the books every round. Thinner books, quicker walk, even though the cook spends a moment reading the shorthand back.
- In the model
- Decode is memory-bound. Fewer bytes per weight means less traffic per token. The kernel dequantizes the weights as it reads them.
04 / What to shrink
Should you shrink only the weights, or the activations too?
Weights are the fixed recipe books. Activations are the numbers flowing through the model for your tokens, the ingredients on the counter. Papers name the choice with a short code: W4A16 means 4-bit weights and 16-bit activations, W8A8 means both in 8 bits. [2]
Weight-only (W4A16, W8A16) saves memory and speeds up memory-bound decode. The multiplications still happen in 16 bits. GPTQ is upfront about it: no speedup "for the actual multiplications, due to the lack of hardware support for mixed-precision operands (e.g. FP16 x INT4)". [1]
Weights plus activations (W8A8) let the GPU multiply in 8 bits. SmoothQuant explains why you'd bother: "to speed up the inference, we need to quantize both weights and activations into INT8", and INT8 for both can "nearly double the throughput of matrix multiplications compared to FP16". [3] NVIDIA introduced FP8 on the H100 for the same reason, "enabling higher throughput of matrix multiplies". [6]
The catch is that activations are harder. They change with every prompt, so their scale is either measured ahead of time on sample text (static) or worked out while the model runs (dynamic). [3] And they're full of outliers, which is chapter 6.
- In the kitchen
- Weight-only: the books are in shorthand, but the cook writes each line out in full grams before weighing anything. W8A8: the cook also weighs on a coarser scale, so the weighing itself is quicker.
- In the model
- W4A16 dequantizes weights to 16 bits for the multiply. W8A8 runs the multiply in 8 bits on hardware that supports it.
05 / The ruler
Who decides the scale, and for how many numbers?
A scale says how big one step is. The most common rule is absmax: find the largest number in the group and make it the top step. In INT8 that top step is 127. [4] Floating-point FP8 needs a scale too. The FP8 paper picks one so the largest value lands near the format's maximum, and it reports that casting to FP8 with no scale at all caused "a significant accuracy loss". [5]
The real choice is how many numbers share one scale. SmoothQuant lays out the options: [3]
- Per-tensor: one scale for a whole matrix. Cheapest, and the most fragile.
- Per-channel: one scale for each output channel of a weight matrix.
- Per-token: one scale for each token's activations.
- Group-wise: one scale per small group of channels. GPTQ and AWQ use groups of 128 weights by default, and GPTQ reports results with them too. [1] [2]
Smaller groups fit better, but every scale costs storage. GPTQ puts groups of 128 at about 0.15 extra bits per weight. [1] Newer hardware builds the idea in: NVIDIA's MXFP8, added with Blackwell, gives "a different scaling factor to each block of 32 consecutive values". [6] And the GPU kernel sets limits: for fast INT8 multiplies, scales can only run along the token and output-channel directions, not per input channel. [3]
Watch what one big number does to everyone sharing its ruler.
Figure 2 · One outlier, eight numbers, one ruler
Format
Scale shared by
Last number
| Value | Original | Stored as | Comes back as |
|---|---|---|---|
| Value 1 | 0.12 | 0 | 0.000lost |
| Value 2 | -0.31 | 0 | 0.000lost |
| Value 3 | 0.27 | 0 | 0.000lost |
| Value 4 | -0.05 | 0 | 0.000lost |
| Value 5 | 0.44 | 0 | 0.000lost |
| Value 6 | -0.18 | 0 | 0.000lost |
| Value 7 | 0.09 | 0 | 0.000lost |
| Value 8outlier | 8.00 | 7 | 8.000 |
INT4, one scale for all 8, with the outlier: each step is 1.143 wide, so all 7 normal numbers come back as 0.
- In the kitchen
- Shorthand counts in spoons, and one note says how big a spoon is. One spoon size for the whole book, one per recipe, or one per page.
- In the model
- Each scale is shared by a tensor, a channel, a token or a group. Finer sharing means less rounding error and more scales to store.
06 / The trap
Why does naive INT8 break on big models?
Because of the problem in Figure 2, at scale. LLM.int8() says it plainly: with one scale per tensor, "a single outlier can reduce the quantization precision of all other values". [4]
And large models have outliers in their activations, in a very particular pattern. The LLM.int8() authors found features up to 20 times larger than the rest. They first appear in some layers, and at around 6.7 billion parameters a "phase shift" spreads them to every layer and to 75% of tokens. Yet they sit in only about 6 feature dimensions, around 0.1% of the inputs. Setting those few to zero made perplexity 600% to 1000% worse. [4] They're rare, and the model depends on them.
SmoothQuant measured the damage. On OPT-175B, the FP16 model averaged 71.6% across four tasks. With INT8 activations on one per-tensor scale, it fell to 32.3%. [3] Its explanation is Figure 2: outliers about 100 times larger than most values stretch the ruler, so ordinary channels get only 2 or 3 effective levels. It also found two helpful facts. The weights are "quite uniform and flat", so they're the easy part. And outliers stay in the same channels for every token. [3]
FP8 copes better, because its steps are fine near zero. In the FP8 paper's tests, both 8-bit formats used the same scaling. BERT Base's F1 score went from 88.19 to 76.89 in INT8 and to 88.09 in E4M3. GPT-3 6.7B's perplexity went from 8.51 to 10.29 in INT8 and to 8.41 in E4M3. [5]
- In the kitchen
- One recipe calls for 8 kg of salt. Size the spoon for that, and every pinch of saffron in the book rounds to zero spoons.
- In the model
- A few activation channels are huge, in every token. A shared scale sized for them rounds the ordinary values away.
07 / The methods
How do GPTQ, AWQ, SmoothQuant and LLM.int8() fix it?
All four are post-training methods: they take a finished model and a little sample text, and need no retraining. They attack different parts of the problem.
| Method | Shrinks | The idea |
|---|---|---|
| LLM.int8() | Weights and activations to INT8 | Keep the outlier dimensions in 16 bits and do the rest in 8. [4] |
| SmoothQuant | Weights and activations to INT8 (W8A8) | Move the outliers' difficulty from activations into weights. [3] |
| GPTQ | Weights only, to 3 or 4 bits | Round one column at a time and nudge the rest to cancel the error. [1] |
| AWQ | Weights only, to 3 or 4 bits | Protect the few weights that meet big activations by scaling them up first. [2] |
LLM.int8() splits each multiply in two. Any feature dimension with a value above 6.0 goes through a 16-bit multiply, and "more than 99.9% of values" go through an 8-bit one. It ran models up to 175 billion parameters "without performance degradation", and cut BLOOM-176B's memory by 1.96 times. It's a memory method first: the split costs speed, and in its matrix tests only the 13B and 175B sizes came out faster. [4]
SmoothQuant divides each activation channel by a factor and multiplies the matching weights by the same factor. The result is "mathematically equivalent", but the outliers are flattened and the weights absorb a little of the difficulty. A setting called α decides how much moves; 0.5 worked for most models it tried. It's done offline, once. In FasterTransformer it reached "up to 1.56× speedup" and halved memory against FP16. [3]
GPTQ quantizes a weight matrix column by column. After each one, it adjusts the weights not yet quantized to make up for the rounding, using how the layer's inputs behave on sample text. That sample was just 128 random 2,048-token pieces of web text. It quantized 175-billion-parameter models "in approximately four GPU hours", to 3 or 4 bits. [1]
AWQ starts from an observation: "Protecting only 1% salient weights can greatly reduce quantization error". The important weights are found by looking at the activations they multiply, not at the weights themselves. Instead of keeping them in 16 bits, AWQ scales those channels up before rounding, so they lose less. It needs no backpropagation or reconstruction, which its authors say helps it avoid overfitting the sample text. [2]
These aren't rivals in practice. vLLM's own INT8 W8A8 recipe runs SmoothQuant first and then GPTQ. [10]
- In the kitchen
- Keep the salt-heavy recipes in full (LLM.int8()). Rebalance the units so no recipe needs a giant spoon (SmoothQuant). Fix each rounding in the next line (GPTQ). Write the recipes people order most in bigger print (AWQ).
- In the model
- Mixed precision, a scale migration, error compensation, and activation-aware scaling. All four run once, after training, on a little sample text.
08 / The cards
Can you shrink the KV cache the same way?
Yes, and it's a separate choice from the weights. Part 1 showed Llama 3 8B needs 128 KiB of cards per token at 16 bits. In FP8 that's 64 KiB. vLLM says this "can significantly reduce its memory footprint", so more tokens fit. [12]
The cards need scales too, and this is where people get caught out. vLLM gives you three levels: [12]
-
No calibration. Set
--kv-cache-dtype fp8and "All quantization scales are set to 1.0." - Calibrated, per tensor. One scale each for queries, keys and values, measured on sample text with the llm-compressor library. vLLM marks calibration as recommended.
- Calibrated, per attention head. One scale per head. It currently needs the Flash Attention backend and llm-compressor's calibration.
A few more details from the same page. fp8 means E4M3, and
fp8_e5m2 is the other option. [14] With the
Flash Attention 3 backend, queries are quantized to FP8 as well, so attention itself runs
in FP8. Some layer types, like sliding-window attention, are "more sensitive", and
--kv-cache-dtype-skip-layers can leave them in full precision.
[12]
- In the kitchen
- The cards in shorthand, with a note on each tray saying how big a step is. Skip the note, and every tray assumes the same step.
- In the model
- FP8 KV cache with per-tensor or per-head scales. Uncalibrated scales default to 1.0.
09 / The bill
How do you know what you lost?
These papers mostly lead with perplexity, how surprised the model is by real text. Lower is better. GPTQ calls it "a very stringent accuracy metric", and AWQ says it "can stably reflect the LLM's performance". [1] [2] It's a good early warning.
But look at what the same papers also checked. LLM.int8() reported zero-shot accuracy on four tasks. GPTQ added LAMBADA, ARC and PIQA. AWQ compared instruction-tuned answers using GPT-4 as a judge, and tested coding and maths. [4] [1] [2] The FP8 paper reported INT8's loss on a question-answering score, not only on perplexity. [5] They measure tasks because tasks are what you ship.
Your sample text matters too. AWQ warns that GPTQ "may overfit the calibration set", and showed the effect by calibrating on medical abstracts and testing on emails. [2] vLLM's guidance is to use calibration data "that closely matches your deployment data", start with 512 samples, and use the model's own chat template. [11]
Two practical traps from vLLM's docs. Quantized models "can be sensitive to the presence
of the bos token", so evaluate with add_bos_token=True.
[9] And if your GPU lacks the kernels for a W8A8 checkpoint,
vLLM falls back to running it as W8A16 and logs a warning. That's "not the scheme the
checkpoint was calibrated and benchmarked with". VLLM_STRICT_QUANT_SCHEME=1
turns the fallback off. [8]
- In the kitchen
- Don't just check that the shorthand books look tidy. Cook the dishes your customers order and taste them against the originals.
- In the model
- Perplexity first, then task benchmarks, then your own prompts, all against the unquantized model with the same settings.
10 / Hands-on
Which format should you pick in vLLM?
Start with your GPU, because native support decides what's fast. NVIDIA's docs say the "H100 GPU introduced support for a new datatype, FP8", and Transformer Engine lists FP8 training on Hopper, Ada, Blackwell and Rubin GPUs. [6] [7] vLLM's docs say the same in its own terms: [9] [10] [11]
| Scheme | Where it runs |
|---|---|
| FP8 W8A8 | Compute capability 8.9 and up: Ada Lovelace, Hopper, Blackwell. On Turing and Ampere, FP8 models run weight-only (W8A16) through Marlin kernels. |
| INT8 W8A8 | Turing, Ampere, Ada Lovelace and Hopper. Not supported on compute capability 10.0 and up, such as Blackwell. |
| INT4 W4A16 | Ampere, Ada Lovelace, Hopper and Blackwell. |
vLLM claims FP8 gives "a 2x reduction in model memory requirements and up to a 1.6x improvement in throughput with minimal impact on accuracy". [9] Its suggested FP8 recipe needs no calibration data at all: per-channel weight scales and per-token activation scales worked out at run time. [9]
Then there are two ways in.
-
A pre-quantized checkpoint. Point vLLM at it. If
--quantizationisn't set, vLLM reads the method from the model's config file. [14] -
Quantize at load time.
--quantization fp8_per_tensororfp8_per_blockconverts a BF16 model's weights as it loads, with no calibration data. [13] For the per-tensor mode, the docs warn that "latency improvements are limited". [9]
vllm serve <model> \
--quantization fp8_per_tensor \
--kv-cache-dtype fp8
vLLM's docs list many more methods, including AWQ, GPTQ, bitsandbytes, GGUF and 4-bit floating-point formats. [8] The list changes often, so check the page for your version.
- On an FP8-capable GPU, start with an FP8 W8A8 checkpoint. It's the smallest step down from BF16, and the GPU can do the maths in 8 bits.
- Go to 4-bit weights (GPTQ or AWQ, groups of 128) when the model doesn't fit, or traffic is low and per-token speed matters most.
- Add an FP8 KV cache when long chats are the limit, with calibrated scales.
- Before switching, run the same task evals on the BF16 model and the quantized one, with calibration data like your traffic.
- Read the startup log. Check which kernel was selected, and that nothing fell back to a different scheme.
Keep these names
A small glossary
- Quantization
- Storing numbers in a smaller format, plus a scale to map them back.
- BF16, FP16, FP8
- Floating-point formats of 16 and 8 bits. FP8 comes as E4M3 (more precision) and E5M2 (more range).
- INT8, INT4
- Evenly spaced whole numbers in 8 or 4 bits, always used with a scale.
- Scale
- The size of one step, shared by a tensor, channel, token or group of values.
- Activations
- The numbers flowing through the model for your tokens, as opposed to its fixed weights.
- W4A16, W8A8
- Shorthand for weight and activation bits: 4-bit weights with 16-bit activations, or both in 8 bits.
- Outlier
- A value far bigger than its neighbours. In large models, a few activation channels are full of them.
- Calibration
- Running sample text through the model to measure the scales a method needs.
- Post-training quantization
- Quantizing a finished model without retraining it. GPTQ, AWQ, SmoothQuant and LLM.int8() all work this way.
- Perplexity
- How surprised a model is by real text. Lower is better.
Check the concept
Can you explain it back?
1. How much memory do a 70-billion-parameter model's weights take in INT4?
About 35 GB: 70 billion numbers at half a byte each, plus a little for scales.
2. Why can 4-bit weights speed up decoding even if the maths stays in 16 bits?
Decode is limited by memory traffic, not maths. Each step reads all the weights, and 4-bit weights are a quarter of the bytes. The kernel expands them as it reads.
3. One activation channel holds values 100 times bigger than the rest. What happens with one INT8 scale for the whole tensor?
The scale is sized for the outlier, so ordinary values get only a few steps and many round to zero. Accuracy collapses.
4. What's the difference between what SmoothQuant and AWQ quantize?
SmoothQuant quantizes weights and activations to 8 bits (W8A8). AWQ quantizes only the weights, to 3 or 4 bits, and leaves activations in 16 bits.
5. You turn on an FP8 KV cache and skip calibration. What scale does vLLM use?
1.0 for every scale. vLLM recommends calibrating with a dataset for the best accuracy.
Continue the thread
The rest of the kitchen
- Part 1: KV cache. What the cards are, how big they get, and why decode is memory-bound.
- Part 2: PagedAttention. How the freed memory is handed out to trays without waste.
- Part 3: Continuous batching. How the host keeps every seat busy, every step.
- Part 4: Speculative decoding. How an apprentice's guesses let the cook write several words per trip.
- Inference Masterclass: the whole kitchen in one place.
- How a GPU works: memory bandwidth, compute, and why the walk to the warehouse dominates.
Read further
Sources
All accessed 18 November 2026. vLLM docs pages are the "latest" version on that day.
- Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", ICLR 2023
- Lin et al., "AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration", MLSys 2024
- Xiao et al., "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models", ICML 2023
- Dettmers et al., "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale", NeurIPS 2022
- Micikevicius et al., "FP8 Formats for Deep Learning", 2022
- NVIDIA Transformer Engine docs: Using FP8 and FP4 with Transformer Engine
- NVIDIA Transformer Engine README (GitHub)
- vLLM docs: Quantization (supported formats, hardware table, activation fallbacks)
- vLLM docs: FP8 W8A8
- vLLM docs: INT8 W8A8
- vLLM docs: INT4 W4A16
- vLLM docs: Quantized KV Cache
- vLLM docs: Online Quantization
- vLLM docs: Engine Arguments
- Meta llama-models: model shapes (sku_list.py)
Written with AI assistance. Every technical claim was checked against the original sources listed above.