Why Better Data Can Beat a Bigger Model
See why clean, representative data often matters more than a bigger model.
When a model underperforms, the tempting fix is to make it bigger: more layers, more weights, more power. Sometimes that’s right. Often it’s like hiring a smarter student and handing them a textbook full of typos.
A model can’t be more right than its examples, and data lets you down in two classic ways:
- Wrong labels. If one cat photo in five is labelled “dog”, the model learns that cats are sometimes dogs. A bigger model makes this worse, not better: it has the freedom to memorise every mistake faithfully.
- Missing cases. Train a pedestrian detector only on sunny daytime footage and it has never seen a person at night. No amount of size invents examples that aren’t there.
The fix is unglamorous: check the labels, remove duplicates, fill the gaps. And it works. In 2023 Microsoft trained a small coding model, phi-1, mostly on carefully chosen “textbook quality” material, and it beat coding models more than ten times its size.
Bigger models really do win when the data is good. They just can’t rescue data that isn’t.
Key takeaways
- A model’s ceiling is its data: it can’t be more right than its examples.
- Wrong labels hurt big, flexible models most, because they can memorise every error.
- Missing cases can’t be fixed by size: the model has nothing to learn them from.
- Cleaning data and filling its gaps is often the cheapest route to a better model.
Check your understanding
Go deeper (more technical)
The “big model” in the simulation copies the label of the nearest training example (a 1-nearest-neighbour classifier). It isn’t a neural network, but it shares the key trait of a huge, flexible one: it fits every example perfectly, wrong labels included. The “small model” can only draw a straight line, so a few bad labels barely move it.
Careful data work now has a name, data-centric AI, popularised by Andrew Ng around 2021. That same year, a study by Northcutt and colleagues found label errors in the test sets of ten popular benchmarks: at least 3.3% on average, and about 6% of ImageNet’s validation set. That’s enough to change which model looks best.
Size still matters. With clean, plentiful data, bigger models keep getting better, which is the story of modern LLMs. The lesson isn’t “small beats big”. It’s “fix the data first”.