A 60-second recap
- A CPU has a few powerful cores built for latency: one complicated task, finished fast.
- A GPU has thousands of simple units built for throughput: the same maths on lots of data at once.
- Work runs in warps of 32 threads that execute the same instruction together.
- The GPU hides slow memory by switching between many warps.
- For AI, memory bandwidth is often the real limit, and Tensor Cores do most of the maths.
Part 2 treated the GPU as "a lot of SMs plus memory". This lesson opens every box.
In the stadium
Part 2 showed the stadium from the outside: many sections of students, a supervisor per row, and a library across the road. Today we walk through every tier, section and desk.
In the silicon
We will name each level of NVIDIA's design (GPC, TPC, SM, processing block) and each unit inside it, using the H100 as the example.
The zoom: from the whole chip to one unit
A modern NVIDIA GPU is built like a set of nesting boxes. The full GH100 chip inside the H100 has:
- 8 GPCs (Graphics Processing Clusters), the biggest building blocks;
- each GPC has 9 TPCs (Texture Processing Clusters);
- each TPC has 2 SMs (Streaming Multiprocessors);
- so the full chip has 8 × 9 × 2 = 144 SMs.
The H100 SXM5 you can buy has 132 SMs switched on (66 TPCs), and the PCIe card has 114. Chips this big are rarely perfect, so some blocks are disabled and every product uses the same enabled count. Click to zoom in:
In the stadium
The stadium has 8 tiers (GPCs). Each tier has pairs of sections that share a stairway (TPCs). Each section (SM) has four rows, each with its own supervisor (processing blocks).
In the silicon
GPU → 8 GPCs → 9 TPCs each → 2 SMs each → 4 processing blocks each. On H100 SXM5: 132 SMs, 528 Tensor Cores, 16,896 FP32 units.
Every unit inside an SM
Zoom into one H100 SM and you find four identical processing blocks (also called partitions), sharing one pool of fast memory. Per SM, NVIDIA lists 128 FP32 units, 64 FP64 units, 64 INT32 units and 4 Tensor Cores, plus 65,536 32-bit registers. Divided by four, each processing block has:
| Unit | Per block | What it does | Stadium version |
|---|---|---|---|
| Warp scheduler | 1 | Every cycle, picks a warp that is ready and issues its next instruction | The row supervisor |
| Register file | 16,384 × 32-bit (64 KB) | Each thread's private working values; the fastest memory on the chip | Each student's scrap paper |
| FP32 units | 32 | Everyday decimal maths (32-bit floating point) | Students doing ordinary sums |
| INT32 units | 16 | Whole-number maths, such as working out memory addresses and loop counters | Students counting and keeping track |
| FP64 units | 16 | High-precision 64-bit maths for science and engineering | The careful accountants |
| Tensor Core | 1 | Multiplies and adds whole small matrices at once (FP8 to FP64) | The times-table machine |
| Load/store units | several | Move data between registers and memory | Runners to the whiteboard and library |
| Special function units | several | Fast approximations for sine, cosine, square roots and reciprocals | A student with a scientific calculator |
NVIDIA's public documents for H100 do not state the exact number of load/store and special function units per block, so we don't print one. Older NVIDIA whitepapers show them per block, and the H100 diagram follows the same pattern.
Shared by all four blocks are the L1 data cache and shared memory (256 KB together), the Tensor Memory Accelerator (Chapter 6) and the texture units.
Why FP64 matters, and why gaming cards cut it
The H100 does FP64 at half its FP32 rate (34 vs 67 TFLOPS), because supercomputing needs it. Gaming GPUs keep only a token amount: NVIDIA's Ada (RTX 40) design runs FP64 at 1/64 of its FP32 rate, "to ensure any programs with FP64 code operate correctly". Same family of ideas, very different balance of units.
The full memory system
Part 2 showed the memory ladder. Here are its real rungs on an H100 SXM5:
| Level | Size | Shared by | Notes |
|---|---|---|---|
| Registers | 256 KB per SM | One thread each | Up to 255 registers per thread |
| L1 + shared memory | 256 KB per SM | Shared memory: one thread block (or a cluster, Chapter 6) | Split is configurable: up to 228 KB can be shared memory |
| L2 cache | 50 MB | All SMs | Full GH100 has 60 MB; connected through a partitioned crossbar |
| HBM3 | 80 GB · 3.35 TB/s | The whole GPU | 5 stacks, 10 × 512-bit memory controllers (full chip: 6 stacks, 12 controllers) |
Shared memory is special: the program decides what goes in it. A common pattern is to load a tile of data from HBM into shared memory once, then let every thread in the block reuse it many times. That is how a GPU raises the "maths per byte" we met in Part 2's roofline chart.
The split between L1 cache and shared memory can be set to 0, 8, 16, 32, 64, 100, 132, 164, 196 or 228 KB of shared memory, with the rest used as cache. One thread block can use up to 227 KB, because CUDA keeps 1 KB per block for itself.
In the stadium
Smart sections copy the pages they need from the library onto their whiteboard once, then everyone in the section reads from the whiteboard instead of running across the road.
In the silicon
Kernels stage data in shared memory so each byte fetched from HBM is reused many times by the threads of a block.
How work lands on an SM
A GPU program launches threads in thread blocks (groups of up to 1,024 threads that can share shared memory). The GPU hands whole blocks to SMs. Each SM can hold several blocks at once, as long as they fit. On H100, each SM can hold at most:
- 64 warps (2,048 threads),
- 32 thread blocks,
- 65,536 registers in total, shared by every resident thread,
- 228 KB of shared memory, shared by the resident blocks.
Whichever runs out first sets how many warps the SM can keep ready. The share of the 64 warp slots in use is called occupancy. Remember Part 2: more ready warps means more chances to hide memory waits. Try it:
Warp slots on one SM
High occupancy is not always the goal. A kernel that keeps lots of data in registers can be fast with fewer warps. But very low occupancy often means the SM sits idle while it waits for memory.
In the stadium
A section has 64 seats for rows of students, limited scrap paper and one whiteboard. Hand out too much scrap paper per student and fewer rows fit, so the supervisor has fewer rows to switch between.
In the silicon
Registers per thread, threads per block and shared memory per block decide how many warps stay resident, and so how well latency can be hidden.
Hopper's new tricks
The H100 (the "Hopper" generation) added features aimed at keeping huge AI workloads fed:
Thread Block Clusters
Before Hopper, thread blocks could only cooperate through slow global memory. A cluster is a new level of grouping: several thread blocks guaranteed to run at the same time on SMs in the same GPC, so they can synchronise and share data directly. The order is now threads → blocks → clusters → grids. A cluster can have up to 8 blocks portably, or 16 on H100 if a program opts in.
Distributed Shared Memory
Blocks in a cluster can read and write each other's shared memory directly, SM to SM. In the stadium: neighbouring sections in the same tier can read each other's whiteboards without going to the library.
Tensor Memory Accelerator (TMA)
A dedicated copy unit inside each SM that moves whole blocks of data (1D up to 5D) between HBM and shared memory. Before, threads had to spend registers and instructions computing addresses and copying data themselves. With TMA, one thread starts the copy and everyone keeps computing.
Asynchronous transaction barriers
A way for threads to wait until a data copy has fully arrived, so copying and computing can overlap safely.
Transformer Engine and FP8
Hardware plus software that picks between 8-bit floating point (FP8) and 16-bit for each layer and handles the rescaling automatically. FP8 comes in two flavours: E4M3 (more precision) and E5M2 (more range).
DPX instructions
Special instructions that speed up dynamic programming algorithms, such as Smith-Waterman for DNA alignment and Floyd-Warshall for shortest paths, which NVIDIA says run up to 7× faster than on A100.
Feeding the chip: copy engines and PCIe
Data usually starts in the CPU's memory and travels to the GPU over PCIe. On H100 that link is PCIe Gen 5: 128 GB/s total, 64 GB/s each way. A one-way copy at 64 GB/s is about 50 times slower than the GPU's own 3.35 TB/s HBM, so copies have to be planned carefully.
GPUs have separate copy engines: hardware that moves data to, from and between GPU memories on its own, while the SMs keep computing. As NVIDIA explains in its guide to overlapping transfers, a GPU with two copy engines can copy data in, copy data out and run a kernel, all at the same time. Programs get this by splitting work into chunks and using separate CUDA streams, with "pinned" (page-locked) CPU memory.
No overlap: copy, compute, copy, repeat
With copy engines: copies run while the SMs compute
copy in (CPU → GPU)compute on SMscopy out (GPU → CPU)
Many GPUs as one
Big models do not fit on one GPU, so GPUs talk to each other constantly. PCIe is far too slow for that, so NVIDIA uses NVLink:
- Each H100 has 18 NVLink 4 links, 900 GB/s in total (both directions). That is about 7 times PCIe Gen 5.
- An 8-GPU HGX / DGX H100 board connects all 8 GPUs through 4 NVSwitch chips, so any GPU can talk to any other at full speed, at the same time.
- The NVLink Switch System was designed to extend this to up to 256 GPUs.
- On the newer Blackwell generation, NVLink 5 gives each GPU 1.8 TB/s, and a GB200 NVL72 rack joins 72 GPUs into one NVLink domain that NVIDIA describes as acting like "a single, massive GPU".
In the stadium
For a huge exam, several stadiums work together, joined by private high-speed tunnels instead of the public road.
In the silicon
NVLink and NVSwitch connect GPUs far faster than PCIe, so a model can be split across many GPUs that exchange results every step.
One GPU as many: MIG
Sometimes the opposite problem: a job is too small for a whole H100. Multi-Instance GPU (MIG), introduced with the A100, splits one GPU into up to 7 isolated instances. Each gets its own SMs, its own slice of L2 cache and memory controllers, and its own memory, so one user cannot slow down or see another.
Hopper also added confidential computing: NVIDIA calls the H100 the first accelerator with confidential computing: data in use stays protected even from the host operating system and hypervisor (it needs a CPU with a matching trusted environment). NVIDIA designed it to extend this protection to MIG instances.
The graphics side, briefly
GPUs were born for graphics, and gaming GPUs still carry units an AI chip does not need:
- RT Cores (since Turing, 2018) speed up ray tracing: finding which triangle a ray of light hits.
- Texture units fetch and filter images to paint onto surfaces.
- ROPs (raster operation units) write the final pixel colours to the screen.
The H100 is built for data centres: it has no display outputs and no RT Cores, and NVIDIA says only two of its TPCs can run graphics shaders at all. Here is how a top gaming card and the H100 compare:
| GeForce RTX 4090 (Ada) | H100 SXM5 (Hopper) | |
|---|---|---|
| SMs | 128 | 132 |
| Memory | 24 GB GDDR6X | 80 GB HBM3, 3.35 TB/s |
| FP64 speed | 1/64 of FP32 | 1/2 of FP32 |
| Ray tracing | An RT Core in every SM | None |
| GPU-to-GPU links | PCIe | NVLink, 900 GB/s |
Where the TFLOPS headline comes from
Spec sheets say things like "67 TFLOPS of FP32". That number is simple arithmetic:
number of SMs × units per SM × operations per unit per clock × clock speed
Each FP32 unit can do one fused multiply-add per clock, which counts as 2 operations. Try it:
Two cautions when reading spec sheets. First, these are peaks: real programs rarely reach them, mostly because of the memory limits from Part 2. Second, Tensor Core numbers are often quoted "with sparsity", a feature that skips zeros in a fixed pattern (the model's weights must first be pruned so 2 of every 4 values are zero), which doubles the headline. Without it, the H100 SXM5 FP16 Tensor figure is about 989 TFLOPS, not 1,979.
Cheat sheet and quiz
Glossary
- GPC
- Graphics Processing Cluster: the largest building block. The full GH100 has 8.
- TPC
- Texture Processing Cluster: a pair of SMs. 9 per GPC on GH100.
- SM
- Streaming Multiprocessor: runs warps. 132 on H100 SXM5.
- Processing block
- One of the four parts of an SM, each with a warp scheduler, registers and maths units.
- FP32 / INT32 / FP64 units
- Maths units for everyday decimals, whole numbers and high-precision decimals.
- Special function unit
- Fast approximations of sine, cosine, square roots and reciprocals.
- Shared memory
- Fast on-chip memory a program controls, shared by the threads of one block.
- Occupancy
- How many of an SM's 64 warp slots are in use.
- Thread block cluster
- Hopper's group of blocks that run together and can share each other's shared memory.
- TMA
- Tensor Memory Accelerator: a copy unit that moves tiles between HBM and shared memory.
- Copy engine
- Hardware that moves data to, from and between GPU memories while the SMs compute.
- NVLink / NVSwitch
- Fast GPU-to-GPU links, and the switch chips that connect many GPUs.
- MIG
- Multi-Instance GPU: split one GPU into up to 7 isolated instances.
- RT Core
- A unit on gaming GPUs that speeds up ray tracing.
- Sparsity
- A Tensor Core feature that skips zeros in a fixed pattern; doubles headline TFLOPS.
Quick quiz
1. Name the levels from the whole chip down to the part with a warp scheduler.
GPU → GPC → TPC → SM → processing block.
2. Why does using more registers per thread sometimes make a kernel slower?
Registers are shared by every resident thread. More per thread means fewer warps fit on the SM, so there are fewer warps to switch to while waiting for memory.
3. What problem does TMA solve?
Moving tiles of data into shared memory used to take threads' registers and instructions. TMA does the copy in dedicated hardware while the threads keep computing.
4. When would you use MIG, and when NVLink?
MIG when one GPU is too big for a job: split it into isolated instances. NVLink when one GPU is too small: connect many GPUs to work as one.
5. A spec sheet says 1,979 TFLOPS "with sparsity". What is the dense figure?
About half: roughly 989 TFLOPS.
Where this fits
Sources
- NVIDIA, NVIDIA Hopper Architecture In-Depth: GPCs, TPCs, SMs, units per SM, L2, HBM3 stacks and controllers, clusters, DSMEM, TMA, Transformer Engine, DPX, NVLink, PCIe Gen 5, graphics-capable TPCs.
- NVIDIA, H100 Tensor Core GPU Architecture whitepaper.
- NVIDIA, Hopper Tuning Guide: registers, shared memory sizes, occupancy limits, cluster sizes, TMA.
- NVIDIA, CUDA Programming Guide: compute capabilities and thread block clusters.
- NVIDIA, CUDA C++ Best Practices Guide and How to Overlap Data Transfers in CUDA C/C++: copy engines and overlapping transfers with compute.
- NVIDIA, H100 product page and datasheet.
- NVIDIA, Introducing NVIDIA HGX H100 and DGX H100: NVSwitch topology.
- NVIDIA, GB200 NVL72: NVLink 5 and the 72-GPU domain.
- NVIDIA, MIG User Guide and supported MIG profiles.
- NVIDIA, Turing architecture whitepaper (RT Cores) and Ada architecture whitepaper (RTX 4090, FP64 rate).
- NVIDIA Developer Forums, How to calculate the Tensor Core FP16 performance of H100.
Written for supraj.dev. Facts checked on 11 October 2026. Where NVIDIA does not publish a detail (such as exact load/store unit counts, or the boost clock on the product page), the article says so.