How a GPU works, and why AI can't live without it

Every time an AI model answers you, millions of small multiplications happen in a fraction of a second. In Part 1 we saw that a model produces its answer one token (a word or piece of a word) at a time, and that each token needs the model's billions of numbers, its weights. Something has to do all that arithmetic, fast, for millions of people.

That something is almost always a GPU, a graphics processing unit. It was built to draw video game graphics. Today it is the engine of AI: training models, and running them.

To see why, we first need to understand the chip you already know, the CPU (central processing unit), and what it was designed for.

A CPU is designed to finish one task, as fast as possible. The time it takes to finish one task is called latency, and CPUs are latency champions. They get there with a few tricks, all of which take up a lot of the chip:

• Big caches. A cache is a small, very fast memory right next to the core that keeps recently used data close, so the core rarely waits for the slower main memory.
• Branch prediction. Programs are full of "if this, then that". The CPU guesses which way each "if" will go and starts work early. When it guesses right, which is most of the time, no time is lost.
• Out-of-order execution. If one instruction is stuck waiting, the CPU looks ahead and runs later instructions that do not depend on it.
• High clock speeds. Each core does each step very quickly.

This is why your laptop feels snappy: opening apps, running a web browser, an operating system juggling many small different jobs. Each job is different, full of decisions, and needs answering now.

Now give the professors a different job: add 1 to each of 8 million numbers. No decisions, no twists, just the same tiny step, millions of times. The professors' clever tricks do not help much here. What would help is more hands.

A GPU makes a different bet. Instead of finishing one task fast, it tries to finish as many tasks as possible per second. That is called throughput.

NVIDIA's own programming guide puts it plainly: the GPU is designed so that more transistors are devoted to data processing rather than data caching and flow control. Fewer clever tricks per core, many more cores.

Each GPU core is slower and simpler than a CPU core. But when the work is thousands of identical little jobs, being wide beats being fast. Try it:

Why did anyone build such a chip? Graphics. A modern screen has millions of pixels, and a game redraws all of them many times a second. Each pixel's colour can be worked out mostly on its own, with the same steps as its neighbours. That is the perfect stadium job.

For years GPUs could only do graphics. Researchers who wanted their raw arithmetic power had to disguise their maths as pictures. In 2006, NVIDIA introduced CUDA, a way to program its GPUs for general computing in a C-like language, starting with the GeForce 8800 generation. Suddenly the stadium could be hired for any job that looked like "the same step on lots of data".

In 2012, a neural network called AlexNet won the ImageNet image-recognition challenge by a wide margin. Its authors trained it on two NVIDIA GTX 580 GPUs with 3 GB of memory each, over five to six days. That result is widely seen as the moment deep learning took off, and GPUs came with it.

Why AI is "stadium work"

A neural network layer does one main thing: it multiplies a list of input numbers by a big grid of weights. That is called a matrix multiplication. And a matrix multiplication is just a huge number of small multiply-then-add steps, most of which do not depend on each other.

For example, one output number of a layer might be: 0.2×w1 + 0.7×w2 + 0.1×w3 + … for thousands of inputs. Every output number can be computed at the same time as every other one. Large language models are mostly stacks of these multiplications, repeated layer after layer.

Let's open up a real data-centre GPU: NVIDIA's H100 (the SXM5 version, launched in 2022 and widely used for AI). Click the parts.

A thread is one small stream of work, like "compute output number 4,217". A GPU program launches thousands or millions of threads. But an SM does not manage them one by one. It handles them in groups of 32, called a warp.

All 32 threads of a warp run the same instruction at the same time, each on its own data. NVIDIA calls this SIMT: single instruction, multiple threads. It is cheap: one instruction is fetched and decoded once, and 32 threads use it.

But there is a catch. What if threads in the same warp disagree at an "if"? Step through it:

Fetching data from main memory is slow compared with doing maths: hundreds of clock cycles. A CPU fights this with big caches and clever prediction, so it rarely has to wait.

A GPU uses a different trick. Each SM keeps many warps loaded at once, each with its own registers already in place. When one warp has to wait for memory, the scheduler simply switches to another warp that is ready, with no setup cost. With enough warps, someone is always doing maths. This is called latency hiding.

This is also why GPUs need lots of work to be fast. Give a GPU only a few threads and there is nobody to switch to: it waits, just like the lonely warp above.

A GPU has a ladder of memories. The closer to the cores, the faster and smaller:

Here is the key idea. For every job, ask: how much maths do I do for each byte I read from memory? That ratio is called arithmetic intensity.

• If you do lots of maths per byte, the cores are the bottleneck. The job is math-bound.
• If you do little maths per byte, the cores finish quickly and then wait for data. The job is memory-bound, no matter how many cores you have.

The tipping point is the chip's peak maths speed divided by its memory speed. Drag the slider, or pick a preset:

This explains something from Part 1. When a chatbot writes an answer for one person, every new token needs the entire model read from HBM, but does only a little maths with each weight. So token generation is memory-bound: the speed of the library corridor, not the number of students, sets the pace. That is why serving systems like vLLM batch many users together: each weight read from memory is reused for many users at once.

Ordinary GPU cores do one multiply-add for one thread per step. In 2017, NVIDIA's Volta generation (the V100) added a new kind of unit: the Tensor Core. A Tensor Core multiplies and adds a whole small block of numbers (a matrix tile) in one operation.

Smaller numbers, more speed

Numbers in a computer have a precision: how many bits are used to store each one. Classic scientific computing uses 64 or 32 bits. AI models turned out to work well with far fewer, and fewer bits mean less memory to read and more maths per second. Each GPU generation has added new, mostly smaller formats to its Tensor Cores:

Halve the bits and you halve the bytes read from memory for the same model, which matters a lot when you are memory-bound. The trade-off is accuracy, which is why choosing a format is a careful engineering decision. We will cover it properly in the quantization part of this series.

GPUs are not "better CPUs". They are a different tool. Here is the honest comparison:

In practice they always work as a team. The CPU is the host: it runs the operating system and your program, loads data, and decides what to do. The GPU is the device: the CPU sends it big blocks of maths to do, and it sends results back. Data travels between them over a connection such as PCIe, and between GPUs over faster links such as NVIDIA's NVLink.

If you have a computer with an NVIDIA GPU and its driver installed, or a free GPU notebook such as Google Colab, you can look at a real GPU in seconds.

nvidia-smi

nvidia-smi --query-gpu=name,memory.total,memory.used,utilization.gpu --format=csv

import torch
print(torch.cuda.is_available())
print(torch.cuda.get_device_name(0))

Watch utilization.gpu while a model runs. If it is low, the GPU is often waiting: for data from the CPU, or for memory. That is the memory wall from Chapter 8, showing up on your own screen.

Glossary

CPU

Central processing unit. Few powerful cores, built for latency: finishing one task fast.

GPU

Graphics processing unit. Thousands of simple arithmetic units, built for throughput.

Latency

How long one task takes from start to finish.

Throughput

How many tasks are finished per second.

SM

Streaming Multiprocessor: one of the identical blocks a GPU is built from. Runs many warps.

Thread

One small stream of work, like computing one output number.

Warp

A group of 32 threads that run the same instruction together (SIMT).

Divergence

When threads in a warp take different branches, so the paths run one after another.

Latency hiding

Keeping many warps ready, so the SM switches to another one while a warp waits for memory.

HBM

High Bandwidth Memory: the GPU's main memory, stacked beside the chip.

Memory bandwidth

How many bytes per second can move between memory and the chip.

Arithmetic intensity

Maths operations per byte read. Low means memory-bound, high means math-bound.

Tensor Core

A unit that multiplies and adds whole small matrix tiles in one operation.

Precision

How many bits store each number (FP32, FP16, BF16, FP8, FP4). Fewer bits: faster, less exact.

Host and device

The CPU (host) runs the program and sends bulk maths to the GPU (device).

CUDA

NVIDIA's platform for programming its GPUs. Its own article is coming next.

Quick quiz

Next in this series

Sources

1. NVIDIA, CUDA C++ Programming Guide (12.9 archive): "The Benefits of Using GPUs", SIMT architecture, hardware multithreading. Current guide: docs.nvidia.com/cuda/cuda-programming-guide.
2. NVIDIA, GPU Performance Background User's Guide: SMs, arithmetic intensity, math-limited vs memory-limited.
3. NVIDIA, Matrix Multiplication Background User's Guide.
4. NVIDIA, H100 Tensor Core GPU Architecture whitepaper and NVIDIA Hopper Architecture In-Depth.
5. NVIDIA, H100 product page and datasheet: memory, bandwidth, TFLOPS (sparsity figures halved for dense).
6. NVIDIA, Hopper Tuning Guide: registers and shared memory per SM.
7. NVIDIA, Programming Tensor Cores in CUDA 9 and Inside Volta.
8. NVIDIA, Ampere Architecture In-Depth (TF32, BF16) and Introducing NVFP4 (Blackwell FP4).
9. NVIDIA, Blackwell architecture and DGX B200 (up to 8 TB/s per GPU).
10. NVIDIA, Fermi Compute Architecture whitepaper: G80 / GeForce 8800 and the introduction of CUDA in 2006.
11. Krizhevsky, Sutskever, Hinton, ImageNet Classification with Deep Convolutional Neural Networks (AlexNet), NeurIPS 2012.
12. Zhong et al., DistServe (2024): decode is limited by memory bandwidth; batching raises arithmetic intensity.
13. AMD, EPYC 9996 product page and 6th Gen EPYC (Venice) announcement, 2026.
14. NVIDIA, nvidia-smi documentation; PyTorch, torch.cuda.

Written for supraj.dev. Every number in this article comes from the sources above and was checked on 11 October 2026. Simplified drawings and toy numbers are labelled as such.