From Neural Networks to Language Models
See that a language model is the same machine you just learned, trained to predict the next word.
You now know the whole recipe. Neurons multiply, add and bend. Layers stack. The loss scores the mistakes, backpropagation hands out the blame, gradient descent nudges the weights. Repeat a few billion times.
Nothing in that recipe mentions language. So how do you get from a km-to-miles converter to something that writes poems? Three steps.
- Turn words into numbers. Neurons only eat numbers, so every word, or piece of a word, gets an ID, and then a list of numbers the network learns along the way.
- Pick a job with endless answers. Hide the next word: “The cat sat on the ___”. The text itself is the answer key, and the internet has trillions of words of it.
- Scale it up. To predict the next word well on every topic, a model has to soak up grammar, facts, style, even some reasoning, because all of those make the next word more predictable. Nobody programs those skills in. They’re simply what lowers the loss.
That’s a large language model: the network you’ve just met, trained on the simplest possible question, at a size that’s hard to imagine. In the simulation you’ll train a tiny one yourself. The next track opens up the real thing.
Key takeaways
- An LLM is built from the parts you’ve learned: neurons, layers, a loss, backpropagation and gradient descent.
- Words are turned into numbers first, because neurons only work with numbers.
- Its job is to predict the next word, and text supplies endless right answers for free.
- At huge scale, getting good at that one job forces a model to pick up grammar, facts and patterns of reasoning.
Check your understanding
Go deeper (more technical)
The simulation’s model is a single layer: one neuron per word in its 15-word vocabulary, each looking only at the previous word. Its totals become probabilities through softmax: raise e to the power of each total, then divide by their sum. It trains with cross-entropy loss (charge −log of the probability given to the real next word) and plain gradient descent.
Real LLMs differ in three big ways. They split text into tokens, often pieces of words. They read thousands of previous tokens at once, not just one, using a design called the transformer. And they’re enormous: GPT-3 (2020) had 175 billion parameters and trained on about 300 billion tokens.
Next-word prediction alone gets you a brilliant autocomplete, not yet a helpful assistant. Turning one into the other takes extra training with human feedback, which Track 2 covers too.