Why Moving Data Can Be Slower Than Computing
Discover that a GPU is often waiting for data, not doing maths, and what decides which one limits it.
So far the story has been “more cores, more sums per second”. Time for the plot twist: for a lot of AI work, the maths units of a GPU spend most of their time doing nothing. They’re waiting for numbers to arrive.
Picture a super-fast chef fed from a storeroom by a conveyor belt. The chef’s speed is the GPU’s compute; the belt’s speed is its memory bandwidth. An H100 can do about 989 trillion operations a second in 16-bit, but its memory delivers “only” 3.35 trillion bytes a second. For the chef to stay busy, every byte that arrives has to be used for roughly 300 operations.
That ratio, operations per byte loaded, is called arithmetic intensity, and it decides everything. High intensity: the chef is busy and the belt keeps up (compute-bound). Low intensity: the chef stands with empty hands (memory-bound).
Now think about a chatbot writing a reply to one person. To produce each token, every weight must travel from memory, and each weight (2 bytes) is used for just 2 operations, one multiply and one add. That’s an intensity of about 1, three hundred times too low. The GPU is more than 99% idle, and the speed of writing is set almost entirely by how fast the weights can stream in.
The fix comes straight from the matrices lesson: batching. Serve 64 people at once and every weight that arrives is used for 64 people’s tokens, so the intensity jumps 64-fold for nearly the same memory traffic. That’s why AI services pack requests together, and why “memory, not maths” is the headline you’ll keep meeting in the rest of this track and in Track 4.
The chart on the right is called a roofline: memory sets the slope on the left, compute sets the flat roof on the right. Your job is to find out where AI workloads land on it.
Key takeaways
- A GPU can do far more maths than its memory can feed it: about 300 operations per byte on an H100.
- Arithmetic intensity = operations per byte loaded. Low intensity means waiting for memory.
- Writing a reply for one user is memory-bound: each weight is loaded and used just once per token.
- Batching many users or examples raises the intensity, which is why AI services process requests together.
Check your understanding
Go deeper (more technical)
Roofline model: attainable FLOP/s = min(peak FLOP/s, bandwidth × intensity). For an H100 SXM (≈989 TFLOP/s dense BF16, 3.35 TB/s HBM3) the “ridge point” is about 989 / 3.35 ≈ 295 FLOPs per byte. Below that, a kernel is memory-bound.
Decoding one token at batch size 1: each BF16 weight (2 bytes) is used for 2 FLOPs, so the intensity is about 1 FLOP/byte. The GPU reaches at most 3.35 TFLOP/s of its 989, about 0.3%. With a batch of B requests, intensity grows roughly to B (until the KV cache traffic starts to dominate).
The memory wall is getting worse: from A100 to H100, peak dense BF16 compute grew about 3.2× (312 → 989 TFLOP/s) while bandwidth grew about 1.7× (2.0 → 3.35 TB/s).
Spec-sheet numbers; real kernels reach somewhat less.