Inside GPU Memory: Registers, Cache and VRAM
Learn the layers of memory in a GPU, and why “close and small” beats “far and big”.
If memory is the bottleneck, why not just build faster memory? Because of a stubborn law of engineering: memory can be big, fast or cheap, but never all three. Fast memory needs lots of transistors per bit and has to sit right next to the workers, where space is precious. So every computer uses a ladder of memories.
On a GPU, from the top of the ladder down:
- Registers: a few numbers per thread, inside the arithmetic units themselves. Instant.
- Shared memory and L1 cache: a small scratchpad next to each group of cores, which the programmer can fill on purpose.
- L2 cache: tens of megabytes shared by the whole chip.
- HBM (VRAM): the GPU’s main memory, tens of gigabytes stacked right beside the chip. This is where the model’s weights live.
Below that, off the GPU entirely, sits the computer’s ordinary memory and its disk, reached through a much narrower connection. A model that doesn’t fit in HBM has to keep shuttling pieces across that link, and slows to a crawl.
The gaps between the rungs are enormous. If reading a register took one second, reading HBM would take minutes, and the SSD would take days. That’s the whole game of fast GPU code: drag data up the ladder once, then use it as many times as possible before it falls back down. The tiling lesson shows the classic trick for doing exactly that.
Key takeaways
- Memory can’t be big, fast and cheap at once, so GPUs use a ladder: registers, shared memory/L1, L2, HBM.
- The model’s weights live in HBM, the GPU’s main memory. If they don’t fit, things get very slow.
- Data from the CPU side (over PCIe) arrives far more slowly than from HBM.
- Fast GPU programs copy data up the ladder once and reuse it as much as possible.
Check your understanding
Go deeper (more technical)
H100 SXM, approximately:
- Registers: 64K × 32-bit = 256 KB per SM, 132 SMs.
- L1 / shared memory: 256 KB per SM combined, up to 228 KB usable as shared memory.
- L2 cache: 50 MB.
- HBM3: 80 GB at about 3.35 TB/s.
- PCIe Gen5 x16 to the CPU: about 64 GB/s per direction; NVLink between GPUs: about 900 GB/s total.
HBM (high-bandwidth memory) gets its speed from stacking memory chips vertically and connecting them to the GPU with thousands of wires through a silicon base layer. It’s expensive and in short supply, one reason AI chips cost what they do. The cycle counts in the simulation and the illustration are rough orders of magnitude; real latencies depend on access patterns and contention.