TOPIC AI
GPUs & VRAM why AI needs special hardware
IN 10 SECONDS
GPUs handle parallel matrix math equations. Graphics Memory (VRAM) stores active model parameters, dictating the maximum model size you can run.
GOTCHA Model parameter count determines VRAM needs. An FP16 7B parameter model requires at least 14GB of VRAM just to load.
HOW MODEL LOADING FLOWS
01 Model load inference engine parses model file parameters weights.
02 VRAM allocation allocates VRAM blocks (e.g. 16GB) to hold model parameter arrays.
03 Matrix math attention calculations execute in parallel across thousands of GPU cores.
04 Overflow swap if model size exceeds VRAM, parameters spill to slow system RAM, degrading speeds.
POKE IT YOURSELF
nvidia-smi — check GPU resource load and VRAM consumption in Linux
Drill this topic →
~65 sec read