Multi-Head Attention: Looking at Language From Different Angles
See why transformers run many attentions side by side, each free to look for a different kind of link.
One attention pass gives each token one blend of the others. But words relate to each other in many ways at once. In “The tired animal didn’t cross the busy street because it was scared”, “it” needs to know what it refers to; “street” wants its describing word, “busy”; “cross” needs to know who’s doing the crossing; and nearly every word benefits from knowing its neighbour. One set of attention weights can’t point at all of those at the same time.
So the transformer runs several attentions in parallel: multi-head attention. Each head has its own learned query, key and value weights, so each develops its own idea of what makes a good match. One head might track the previous word, another link pronouns to nouns, another connect verbs to their subjects. Their results are combined and added to the token’s vector, so a single layer gathers several kinds of context at once.
The splitting costs nothing extra. Each token’s vector is cut into slices and every head works on its own slice: GPT-3’s vectors of 12,288 numbers are shared among 96 heads of 128 numbers each. Same total work, many more perspectives.
Nobody assigns the jobs. They emerge during training, and researchers then go hunting for them. Some heads turn out to be strikingly specific, such as “previous-token” heads, or “induction” heads that spot a repeated pattern and predict it will continue, which helps a model copy names and phrases from earlier in the text. Many others do something useful that nobody has managed to put into words yet.
Key takeaways
- One attention pass produces one blend per token, but language has many relationships at once.
- Multi-head attention runs several attentions in parallel, each with its own learned weights.
- Heads work on slices of the vector, so many heads cost about the same as one big one.
- Head “jobs” are learned, not assigned. Some are clear (previous-token, induction heads); many aren’t.
Check your understanding
Go deeper (more technical)
Each head computes softmax(Q·Kᵀ / √d) · V on its own slice; the outputs are glued back together and multiplied by one more learned matrix (the output projection) before being added to the residual stream.
Induction heads were described by Anthropic researchers (Olsson and colleagues, 2022), who linked them to in-context learning: the ability to pick up a pattern from examples in the prompt. To save memory, many modern models use grouped-query attention, where several query heads share one set of keys and values: Llama 3 8B has 32 query heads but only 8 key/value heads. Track 4 shows why that matters for serving.