KV Cache: Why AI Doesn't Recalculate Everything
Learn the trick that makes long answers fast, and why it turns memory into the scarcest resource on the GPU.
Back in Track 2, attention worked like this: to write the next token, it compares the new token’s query with the keys of every earlier token, then mixes in their values. That raises an awkward question. When the model writes token 501, does it recompute the keys and values of all 500 tokens before it?
It would be pointless. Earlier tokens never change, and in a language model each token only looks backwards, so their keys and values come out identical every time. So engines compute each token’s keys and values once, the moment the token appears, and keep them. That store is the KV cache.
The saving is enormous. Without the cache, every new token would redo the work for the whole text so far, and long answers would get slower and slower. With it, each step only processes the newest token, then reads everyone else’s cached keys and values.
The price is memory, and plenty of it. Every token of every conversation keeps its keys and values for every layer of the model. For Llama 3 8B that’s about 128 KB per token. A single 128,000-token conversation needs around 17 GB, more than the model’s own weights; a hundred users with ordinary 4,000-token chats need over 50 GB.
That flips the picture from the batching lessons. What usually stops a GPU from serving more people at once isn’t maths, and often isn’t even the weights. It’s room for the KV cache. How well an engine manages that room decides how many users fit, which is exactly what the next lesson is about.
Key takeaways
- Keys and values of earlier tokens never change, so they are computed once and stored: the KV cache.
- With the cache, each decode step only processes the newest token.
- The cache grows with every token of every conversation: about 128 KB per token for Llama 3 8B.
- KV-cache memory, not maths, usually limits how many users a GPU can serve at once.
Check your understanding
Go deeper (more technical)
Cache size per token = 2 (K and V) × layers × KV heads × head dimension × bytes. For Llama 3 8B in BF16: 2 × 32 × 8 × 128 × 2 = 128 KiB per token, so a full 128K-token context needs about 17 GB.
Grouped-query attention (Track 2) shares each key/value head between several query heads (Llama 3: 32 query heads, 8 KV heads), cutting the cache 4×. Engines also store the cache in FP8, and some models compress it further (DeepSeek’s multi-head latent attention).
The work in the simulation counts key/value computations only. Without a cache the total grows as n²/2; with it, as n. Attention still has to read the whole cache every step, which is why very long contexts remain slow.