How AI Represents Meaning as Vectors
Understand how similarity and meaning become geometry: angles and directions between vectors.
Last lesson ended with a table full of vectors. Now for the fun part: what can you actually do with them?
Think of each vector as an arrow pointing somewhere in space. Not 2-D or 3-D space, but space with thousands of directions: Llama 3 8B uses 4,096 numbers per token. Nobody can picture that, but the maths works exactly as it does on paper, and two simple ideas carry straight over.
Similarity is an angle. Arrows pointing the same way belong to related words; arrows at right angles have nothing in common; opposite arrows are opposites. The standard measure, cosine similarity, is simply the cosine of that angle: 1 for “same direction”, 0 for “unrelated”, −1 for “opposite”. Search engines, recommendation systems and the document-search systems in Track 6 all rank things this way.
Directions carry meaning. The step from “man” to “woman” points in roughly the same direction as the step from “king” to “queen”, or from “uncle” to “aunt”. Nobody told the model about gender; the direction emerged because those words differ in the same consistent way across millions of sentences. That’s what makes the famous word arithmetic work: king − man + woman lands near queen.
One honest caveat: neat, nameable directions are the exception. Most of the space doesn’t line up with anything a person would put a label on, and analogies work far less reliably than the famous example suggests. Still, “meaning as position” is the foundation the rest of this track builds on.
Key takeaways
- Embeddings are arrows in a space with thousands of directions.
- Similarity is an angle: cosine similarity is 1 for the same direction, 0 for unrelated, −1 for opposite.
- Consistent differences between words become directions, so “king − man + woman” lands near “queen”.
- Neat directions are the exception: much of the space has no human-readable meaning.
Check your understanding
Go deeper (more technical)
Cosine similarity = (a · b) / (|a| × |b|): the dot product of the two vectors divided by their lengths. The simulation uses plain distance on a 2-D map, which tells the same story for its hand-placed words.
The analogy result became famous with word2vec (2013). Later work pointed out a quiet trick: the input words themselves are excluded from the answer. Without that, the closest vector to king − man + woman is often “king” itself (Nissim, van Noord and van der Goot, 2020).
Inside an LLM, the starting embedding is only the first draft. Every block of the transformer updates each token’s vector using its context, so by the middle layers “bank” near “river” and “bank” near “account” point in clearly different directions.