Prefill vs Decode: The Two Phases of LLM Inference
Understand why reading the prompt and writing the answer stress the GPU in completely different ways.
Every request goes through two phases on the GPU, and they could hardly be more different.
Prefill handles your prompt. All of its tokens are already known, so they can go through the model together in one pass, as one big matrix multiplication. Remember the memory wall from Track 3? The weights are fetched from memory once and each one is used for every prompt token: thousands of operations per byte. The maths units are flat out. Prefill is compute-bound.
Decode writes the answer, and it can’t be done in parallel: token 2 depends on token 1, which depends on the prompt. So each pass produces just one token for this request. But one pass still has to drag every weight out of memory: 16 GB for an 8B model, all of it, for a single token. Each byte is used for about one operation. The maths units mostly wait. Decode is memory-bound.
The numbers are striking. On an H100 with an 8B model, a decode pass takes about 5 milliseconds, set purely by memory speed. Prefill can process roughly 300 prompt tokens in that same time before maths becomes the limit. Reading a 2,000-token prompt costs about as much as writing just seven tokens of the answer.
This split explains a lot of what follows. Decode’s idle maths units are the opportunity that batching fills. Prefill’s heavy bursts can stall other users’ decoding, so engines interleave them carefully. And the two phases are measured separately: time to the first word mostly reflects prefill and queueing, while the speed of the words after that reflects decode.
Key takeaways
- Prefill reads the whole prompt in one parallel pass: compute-bound.
- Decode writes one token per pass and rereads all the weights each time: memory-bound.
- On an H100 with an 8B model, a 2,000-token prefill costs about as much time as seven decode steps.
- Decode’s idle maths units are the opportunity that batching exploits.
Check your understanding
Go deeper (more technical)
With N parameters in 16-bit, one pass reads about 2N bytes of weights and does 2N FLOPs per token processed. For prefill of P tokens the intensity is about P FLOPs per byte; for decode at batch size 1 it is about 1. On an H100, the ridge point is around 295 FLOPs per byte, so prompts longer than a few hundred tokens are firmly compute-bound.
Some systems go further and run prefill and decode on separate GPUs (“disaggregated serving”, for example DistServe, 2024), passing the KV cache across, so heavy prefills never slow down anyone’s stream of tokens.
The simulation uses an 8B model in BF16 on an H100 at spec-sheet peaks and ignores KV-cache traffic.