TOPIC AI
Quantization shrinking models to fit
IN 10 SECONDS
Quantization compresses model parameter weights by converting high-precision numbers (FP32/FP16) to lower-precision formats (INT8/INT4), shrinking file sizes.
GOTCHA Compressing weights too far (e.g. down to 2-bit) breaks connection representations, causing models to produce garbage outputs.
HOW QUANTIZATION COMPRESSES WEIGHTS
01 Base weights large model weights are stored in floating point formats (16-bit).
02 Quantization run compiles floating point scales to integer scales (e.g. 4-bit INT4).
03 Shrink write writes compressed checkpoints, shrinking files by up to 75%.
04 Execution model runs with low VRAM footprint, losing minimal intelligence accuracy.
POKE IT YOURSELF
ollama show llama3 | grep -i quantization — check quantization precision of a local model
Drill this topic →
~65 sec read