Testing AI on Data It Has Never Seen
Know why models are judged on data they never trained on, and how a leak can fake a great score.
You can’t grade a student with the exact questions they practised on. The only honest exam is one they’ve never seen. Models are no different, so before training starts, the data gets split three ways:
- The training set is what the model learns from.
- The validation set is for your choices: which model, how flexible, when to stop. You peek at it a lot.
- The test set stays locked in a drawer. It comes out once, at the very end, for the final verdict.
Why two hidden piles? Because every time you pick something for scoring well, you tune a little to that data. Try fifty versions and keep the one that does best on the validation set, and its score is now flattering. The test set is the one pile no decision ever touched.
The sneakiest failure is data leakage: test examples that secretly also sit in the training data. Duplicate photos. The same patient’s scans on both sides of the split. The model recognises instead of reasoning, scores brilliantly, and fails in the real world. A suspiciously good score deserves suspicion.
Key takeaways
- A score on the training examples flatters a model: it has already seen the answers.
- Split the data: a training set to learn from, a validation set to make choices with, a test set for one final check.
- Choosing by a score slowly tunes you to that data, which is why the test set stays untouched until the end.
- Leakage, when test examples also hide in the training data, makes a model look far better than it is.
Check your understanding
Go deeper (more technical)
The 80/10/10 split is a rule of thumb. With small datasets, people use cross-validation: split the data into, say, 5 parts, train 5 times, each time holding out a different part, and average the scores.
Leaks come in many shapes: exact duplicates, near-duplicates (the same photo, cropped), the same person or patient on both sides of the split, or information from the future sneaking into a forecast (training on data from after the date you’re trying to predict).
Language models have a modern version: benchmark contamination. Test questions published online can end up in the training text scraped from the web, so a model may have seen the exam before. That’s why researchers keep building fresh, unpublished tests.