Why Transformers Replaced Older Language Models
Understand what held older language models back, and the two problems the transformer solved.
Before 2017, the best language models were recurrent neural networks, RNNs. They read the way you’d expect: one word at a time, left to right. After each word they updated a single memory vector, a running summary of everything so far, and passed it on to the next step.
That design has two weaknesses, and both get worse as texts get longer.
- The note fades. Everything about the beginning has to survive being squeezed through the same fixed-size summary, word after word. In “The keys to the old wooden cabinet in my grandmother’s dusty hallway are…”, the model must still remember “keys” (plural) a dozen words later. Improved RNNs called LSTMs (1997) helped a lot, but long gaps stayed hard.
- No shortcuts. Step 50 can’t start until step 49 is done. A GPU has thousands of cores that love doing things at once, and an RNN keeps almost all of them waiting.
In 2017, a team at Google published “Attention Is All You Need” and scrapped the relay race. Their transformer reads all the words at the same time, and every word can look directly at any other word, however far away, through a mechanism called attention. “Are” doesn’t need a fading note about “keys”; it simply looks back at it.
Because every position is processed in parallel during training, transformers could finally use GPUs to the full. That made it practical to train on vastly more text, which, as Track 1 showed, was exactly what was needed. Nearly every major language model since is a transformer.
Key takeaways
- RNNs read one word at a time and squeeze everything into one memory vector.
- Information from early words fades over long gaps.
- Each step waits for the previous one, so RNNs can’t use a GPU’s parallel power during training.
- Transformers read all tokens at once, and attention lets any word look directly at any other.
Check your understanding
Go deeper (more technical)
The simple RNN dates from Elman (1990); the LSTM (Hochreiter and Schmidhuber, 1997) added gates that decide what to keep and what to forget. Attention was actually invented on top of RNNs, for translation (Bahdanau and colleagues, 2014). The transformer’s bold move was to drop the recurrence and keep only attention.
It isn’t free. Attention compares every token with every other, so its cost grows with the square of the length, and generating text is still one token at a time. That’s why RNN-like ideas have made a comeback in state-space models such as Mamba (2023), which aim to combine fast training with cheap long inputs.