The Secret Behind LLMs: Predicting the Next Token
Understand the one simple job every LLM does, over and over: guess the next token.
Your phone’s keyboard has been doing a tiny version of ChatGPT’s job for years. Type “See you” and it offers “tomorrow”, “soon”, “later”. It’s guessing the next word.
That really is all an LLM does at each step. It doesn’t fetch a stored answer, and it doesn’t plan the whole reply first. It reads the text so far and produces a probability for every token in its vocabulary: “Paris” 62%, “the” 21%, “France” 6%, and so on down a list of over a hundred thousand options. One gets picked, it’s glued onto the text, and the whole thing runs again on the slightly longer text.
So where does the cleverness come from? From how good those probabilities are. To guess what follows “The capital of France is”, you need some geography. To continue a half-written function, you need to understand the code. Predicting the next word well turns out to require a startling amount of knowledge, and that’s exactly what training pours into the model.
On the right is a deliberately tiny model that learned from twelve sentences by counting which word followed which. Next to ChatGPT it’s a toddler, but the loop is identical.
Key takeaways
- An LLM writes by repeating one step: predict the next token, pick one, append it.
- Each step produces a probability for every token in its vocabulary, often more than 100,000 of them.
- Good predictions need knowledge, which is why next-word training makes models knowledgeable.
- The toy model only sees the previous word; a real LLM reads the whole conversation at every step.
Check your understanding
Go deeper (more technical)
Formally, a language model learns P(next token | all previous tokens). Generation is autoregressive: each new token is appended to the input for the next step. The toy model here is a bigram model (it only conditions on one previous word); n-gram models like this were state of the art for decades, until neural networks took over.
Training an LLM means adjusting its weights so the probability it gives the actual next token in its training text goes up. It’s the same “nudge the weights to reduce the loss” loop from Track 1, with the loss called cross-entropy: −log of the probability given to the right token.