Attention: How AI Connects Words Across a Sentence
Understand attention: how each token gathers information from the tokens that matter for it.
Read this: “The animal didn’t cross the street because it was too tired.” What’s “it”? The animal, obviously. Now change one word: “…because it was too wide.” Now “it” is the street. You resolved that without noticing, by letting “it” check the other words for the best fit.
Attention is that move, done with numbers. From its vector, every token computes three things using learned weights: a query (“what am I looking for?”), a key (“what do I contain?”) and a value (“what will I pass on if you pick me?”). Then:
- Score. Compare the token’s query with every other token’s key. A good match scores high: “it” looking for “something that gets tired” matches “animal” better than “street”.
- Share out. A softmax turns the scores into percentages that add up to 100%.
- Mix. Blend the other tokens’ values in those proportions and add the blend to the token’s vector.
Afterwards, the vector for “it” no longer just means “some pronoun”; it carries a good dose of “animal”. Every token does this at the same time, and in a language model each may only look at tokens to its left, because the future hasn’t been written yet.
Nothing here is hand-coded about pronouns or grammar. What counts as a good match is learned, purely because getting it right makes the next token easier to predict. That’s the idea behind the 2017 paper’s cheeky title: “Attention Is All You Need”.
Key takeaways
- Attention lets every token look at the other tokens and decide which ones matter.
- Each token has a query, a key and a value; query–key matches become scores.
- Softmax turns the scores into percentages, and the values are mixed in those proportions.
- In a language model, tokens may only attend to earlier tokens: the future isn’t written yet.
Check your understanding
Go deeper (more technical)
With the queries, keys and values stacked into matrices Q, K and V: Attention(Q, K, V) = softmax(Q·Kᵀ / √d) · V. Dividing by √d (the size of each key) keeps the scores from getting so large that softmax puts 100% on a single token.
The “only look left” rule is a causal mask: scores for future tokens are set to −∞ before the softmax, so they get 0%. Comparing every token with every other costs work proportional to the length squared, one reason long contexts are expensive; FlashAttention (2022) speeds this up on GPUs with the tiling trick from Track 3.