Speculative Decoding: Can a Small Model Speed Up a Big One?
See how a small, fast model can make a big model write faster, without changing a single word it writes.
Decode has a frustrating property you now know well: a pass of the big model costs about the same whether it processes one token or several, because the time goes on reading the weights. Yet each pass only produces one new token. What if we could hand the big model several tokens to process per pass?
The obstacle is that the big model doesn’t know its future tokens yet. speculative decoding gets around that with a cheeky trick: guess them, using a much smaller, faster model.
- Draft. The small model quickly writes the next few tokens, say four.
- Verify. The big model checks all four in a single pass, just like a mini prefill, asking at each position: is this the token I would have chosen?
- Keep and correct. Accept the guesses up to the first mistake, and add the big model’s own token at that point.
If the draft got three right, one big pass produced four tokens instead of one. If it got none right, you still get the big model’s one token, and you’ve lost only the small model’s quick guesses.
The best part: the result is exactly what the big model would have written on its own. The big model has the final say on every token, so speed goes up and quality doesn’t budge. How much faster depends on how often the small model guesses right. Predictable text such as code, common phrases or repeated passages speeds up a lot; surprising text much less. Real systems typically report speed-ups of two to three times.
Key takeaways
- A big-model decode pass costs about the same for one token or several, because it is memory-bound.
- A small draft model guesses ahead; the big model checks all the guesses in one pass.
- Guesses are kept up to the first mistake, plus one token from the big model, so the text is unchanged.
- The speed-up depends on how often the draft is right: typically 2–3× in practice.
Check your understanding
Go deeper (more technical)
Speculative decoding was published independently by Leviathan and colleagues (Google) and Chen and colleagues (DeepMind) in 2022–2023. With sampling (temperature above 0), a rejection-sampling rule accepts drafts so that the output distribution exactly matches the big model’s.
Variants avoid a separate draft model: Medusa and EAGLE add small prediction heads to the big model itself, and prompt lookup drafts by copying text from the prompt, which works well for editing and summarising. At very large batch sizes the GPU is no longer idle, so the gains shrink.
The simulation’s timings (20 ms per big pass, 2 ms per drafted token) are illustrative.