From Words to Numbers: Understanding Embeddings
See how each token becomes a list of numbers, and how those numbers come to capture meaning.
A token ID like #3797 is just a name tag. It says nothing about what “cat” means: #3797 is no closer to “kitten” than to “carburettor”. So the model’s very first move is to swap each ID for an embedding: a vector, a list of a few thousand numbers, looked up in a big table.
Nobody writes those numbers by hand. They start out random and get nudged during training, like every other weight. And yet they end up capturing meaning, thanks to a simple fact about language: words that mean similar things keep similar company. “Cat” and “kitten” both purr, get fed and nap in the sun. “Pizza” and “soup” both get eaten and turn out too hot. As the linguist J. R. Firth put it in 1957, “You shall know a word by the company it keeps.”
Training exploits that. If swapping “cat” for “kitten” barely changes what usually comes next, the model does better by giving them similar vectors, so the same downstream knowledge works for both. Repeat over trillions of words and the table organises itself: related words cluster together, unrelated ones drift apart.
It also means you can work out a word you’ve never seen. If someone says “the wug was delicious, but far too hot”, you already suspect a wug is food. On the right, you’ll build word vectors purely from the blanks they fill, and test exactly that.
Key takeaways
- An embedding is a list of numbers that stands for a token. The first layer of an LLM is a big lookup table of them.
- The numbers are learned during training, not written by people.
- Words used in similar contexts end up with similar vectors: “you shall know a word by the company it keeps”.
- That’s why a model can guess what a rare or new word means from the words around it.
Check your understanding
Go deeper (more technical)
The embedding layer is a big table of size vocabulary × width: Llama 3 8B has 128,256 rows of 4,096 numbers, about 0.5 billion weights. Looking up a row is the same as multiplying a one-hot vector by the table.
The simulation builds vectors by counting contexts, a real technique from the 1990s and 2000s (“count-based” methods such as latent semantic analysis). In 2013, word2vec showed that vectors learned by predicting neighbouring words work even better. Inside an LLM, embeddings are learned as a side effect of predicting the next token.
“Wug” comes from Jean Berko Gleason’s famous 1958 experiment, in which children happily said “two wugs” about a creature they had never seen: people generalise from context and patterns, and so do models.