TOPIC AI
Inference vs Training using vs building a model
IN 10 SECONDS
Training is the process of adjusting weights over vast corpora (requires massive clusters). Inference is using that trained model to generate predictions (requires low latency).
GOTCHA Running training code on inference hardware is extremely slow. Training requires high memory bandwidth, whereas inference prioritizes query latency.
HOW COMPUTE PATHS SEPARATE
01 Training run processes gigabytes of texts, updating model weights using backpropagation.
02 Weights save freezes weights into static parameter files (checkpoints).
03 Inference deploy loads static weights into server VRAM.
04 Generation query receives user query, running forward pass to return next token predictions.
POKE IT YOURSELF
ollama run llama3 — execute model inference locally on your CPU/GPU
Drill this topic →
~65 sec read