How Thousands of GPU Threads Work Together
Understand how work is organised on a GPU: threads, warps, blocks and SMs, and the two ways to waste them.
Thousands of cores need some organising. How does a GPU actually hand out the work?
A GPU program is called a kernel. When it runs, thousands of copies of it start at once, each a thread responsible for one small piece of the result, for example one output number of a matrix multiplication. The threads are organised like a well-run factory:
- Warp: 32 threads that execute the same instruction at the same moment, like rowers in a boat.
- Block: a group of warps that work together and can share a scratchpad of fast memory.
- SM (streaming multiprocessor): a workshop that runs blocks. An H100 has 132 of them.
Two classic mistakes waste this factory.
Idle workshops. Work is split into blocks and dealt out to the SMs in rounds. If you have 133 blocks and 132 SMs, the first round fills every SM, and the second round runs a single block while 131 workshops sit idle. Engineers call this the tail effect.
Rowers pulling in different directions. Because a warp shares one instruction, an “if” where half the threads go one way and half the other makes the warp run both paths, one after the other, with half its threads waiting each time.
Neural network maths is gentle on both counts: huge, evenly divisible piles of work, and almost no “ifs”. Every thread does the same multiply-and-add on different numbers, which is exactly what the warp was designed for.
Key takeaways
- Thread = one small piece of work. Warp = 32 threads moving in lockstep. Block = a group of warps on one SM.
- If the work doesn’t split evenly over the SMs, the last round leaves workshops idle (the tail effect).
- If threads in a warp take different branches of an “if”, the warp runs both branches one after the other.
- Neural network maths has almost no “ifs” and splits evenly, which is one more reason it suits GPUs.
Check your understanding
Go deeper (more technical)
In CUDA, you launch a grid of blocks; each block can have up to 1,024 threads. An SM can hold several blocks at once (up to 2,048 resident threads on H100) and switches between warps every cycle to hide memory delays: while one warp waits for data, another computes. How many fit depends on how many registers and how much shared memory each block uses (“occupancy”).
The tail effect is also called wave quantisation: a matrix multiply split into 133 tiles on a 132-SM GPU needs two waves, the second almost empty. Library authors pick tile sizes partly to avoid this.