Why Word Order Matters
See why a pile of word vectors loses who-did-what-to-whom, and how models stamp tokens with their position.
“Dog bites man” is a Tuesday. “Man bites dog” is front-page news. Same three words, opposite stories. Order isn’t decoration in language; it’s half the meaning.
That’s a problem for a model built from embeddings, because a pile of vectors has no order. Combine the three word vectors, say by averaging them, and both sentences produce exactly the same numbers: the model couldn’t tell who bit whom. And as the next lesson shows, a transformer reads all the words of a text at once rather than one after another, so it can’t pick up the order just by reading in sequence.
The fix is to stamp every token with its position. The original 2017 transformer added a positional encoding: a second vector, unique to each position and built from waves of different speeds, added onto the word’s embedding. Now “dog” in position 1 and “dog” in position 3 are slightly different vectors, and the two sentences no longer look alike.
Most modern models, Llama included, use a neater trick called rotary position embedding (RoPE). Instead of adding a stamp, it rotates parts of each vector by an angle that grows with the position, like the hand of a clock. When the model compares two tokens, only the difference between their angles survives, so what it really sees is how far apart they are, not their absolute address. “The cat” relates the same way on page 1 as on page 300.
Key takeaways
- Word order carries meaning, but a set of vectors has no order of its own.
- Transformers read all tokens at once, so position information has to be added explicitly.
- The original transformer added a position vector to each embedding.
- Modern models rotate vectors by position (RoPE), so comparisons depend on how far apart tokens are.
Check your understanding
Go deeper (more technical)
The original sinusoidal encoding: PE(pos, 2i) = sin(pos / 10000^(2i/d)) and PE(pos, 2i+1) = cos(…). Each pair of dimensions is a wave with its own speed, so every position gets a unique pattern. GPT-2 instead learned a position vector for each of its 1,024 positions.
RoPE (Su and colleagues, 2021) rotates each pair of numbers in the query and key vectors by an angle proportional to the position, with a different speed per pair. Because attention compares queries with keys through a dot product, only the relative rotation survives. Stretching those angles (“RoPE scaling”) is one way models are extended to longer context windows after training.