Batching: How One GPU Handles Multiple Users
See how one GPU serves many users with a single pass, and the flaw in the simplest way of doing it.
The last two lessons pointed to the same opportunity: during decode the GPU fetches every weight from memory and then barely uses it. Batching is how serving systems cash in.
Instead of running one user’s next token, the GPU runs a step for several users at once. Their current tokens are stacked into a little matrix, and the layer becomes the grid-times-grid multiplication from Track 3. The weights are still fetched once, but now each one is used for every user in the batch. Four users cost barely more than one; you get four tokens for about the price of one.
The simplest version is static batching: collect a group of requests, run them together until they’re all done, then start the next group. It works beautifully when every request is the same size, which is how batching is usually done in training. Chat traffic isn’t like that. One person asks for a haiku, the next for a 2,000-word essay.
In a static batch, the haiku finishes in 20 steps, but its seat stays reserved until the essay is done. The GPU keeps doing a step for that empty seat, and new users who arrived meanwhile wait outside until the whole batch ends. With a realistic mix of short and long answers, most of the batch’s capacity can be wasted on seats nobody is using.
So batching is clearly right, but batching like this is clumsy. The next lesson fixes the seating plan.
Key takeaways
- Batching runs one decode step for many users at once, reusing every fetched weight.
- Because decode is memory-bound, a batch of several users costs barely more than one.
- Static batching runs a fixed group until its slowest member finishes.
- With mixed answer lengths, static batches waste most of their seats and make newcomers wait.
Check your understanding
Go deeper (more technical)
A batched decode step multiplies a (B × d) matrix of the batch’s current token vectors by each (d × d) weight matrix, so the arithmetic intensity grows roughly with B, exactly the roofline effect from Track 3.
Prompts of different lengths also had to be padded to the same length in older systems, wasting even more work. Modern engines avoid padding by packing tokens from different requests into one flat sequence and keeping track of where each request starts.
The simulation’s step times are illustrative.