Scaling Laws: Why Bigger Models Get Smarter, Until They Don't
Understand why bigger models got predictably better, how to split a budget between size and data, and where it stops.
For years, building a better language model mostly meant one thing: make it bigger. It’s worth asking why that worked so reliably, and why it can’t go on forever.
In 2020, researchers at OpenAI trained many models of different sizes and found something remarkable: the loss fell along a smooth, predictable curve as they scaled up parameters, data and compute. Each tenfold increase in compute bought roughly the same improvement as the one before. These scaling laws turned AI progress into something you could plan: spend this much, expect roughly that much better.
In 2022, a DeepMind team asked the obvious follow-up. With a fixed compute budget, how should you split it between a bigger model and more training text? Their answer: grow both together, at about 20 training tokens per parameter. By that yardstick, GPT-3 (175 billion parameters, 300 billion tokens) had been far too big for its data. Their own model, Chinchilla, used 70 billion parameters and 1.4 trillion tokens, and beat a 280-billion-parameter model trained with the same compute.
Real teams don’t always follow that ratio. Llama 3 8B was trained on about 15 trillion tokens, nearly 2,000 per parameter, far past the “optimal” point, and that’s deliberate: training is paid once, but a smaller model is cheaper every single time it answers.
And the “until they don’t” part? The curves bend towards a floor, because some of language is genuinely unpredictable: no model can know for sure what the next line of a novel will be. Meanwhile each step down the curve costs about ten times more than the last, and good text is running out. That’s why progress now also comes from better data, smarter training and spending more compute while answering, as the reasoning-models lesson will show.
Key takeaways
- Loss falls smoothly and predictably as models, data and compute scale up: these are scaling laws.
- For a fixed budget, grow model size and data together: roughly 20 tokens per parameter (Chinchilla).
- Smaller models trained for far longer are popular because they are cheaper to run.
- Returns diminish: each improvement costs about 10× more compute, and there is a floor no model can beat.
Check your understanding
Go deeper (more technical)
The simulation uses the Chinchilla form L = E + A/N^α + B/D^β (N = parameters, D = tokens), with the corrected constants from Epoch AI’s 2024 replication of the paper, and compute ≈ 6 × N × D FLOPs (Track 1’s “6 operations per parameter per token”). The first scaling-law paper was Kaplan and colleagues (2020); the Chinchilla paper is Hoffmann and colleagues (2022).
Loss is not the same as usefulness: some abilities seem to appear suddenly at scale, though researchers debate how much of that is an artefact of how they are measured. Studies of the available stock of high-quality public text suggest it could be largely used up within the next few years, which is pushing work on synthetic data and on scaling compute at answer time instead.