When AI Memorizes Instead of Learning
Spot overfitting: a model that aces its training examples by memorising them, then flops on anything new.
Every class has one: the student who memorises last year’s exam word for word. Same questions? Top marks. Change one number? Total collapse. They learned the answers, not the subject.
Models do this too, and it has a name: overfitting. Give a model enough freedom (enough weights) and it can bend itself through every training example, random noise included: the house that sold cheap because the owner was in a hurry, the email someone flagged as spam by accident. Its error on the training examples drops to zero. It looks like a genius.
Then a new example arrives and the genius falls apart. The quirks it memorised don’t repeat, so its predictions go wild.
The opposite failure is underfitting: a model too stiff to capture even the real pattern, like tracing a curve with a ruler. The sweet spot lies in between: flexible enough to catch the pattern, too stiff to chase the noise.
The tell-tale sign of overfitting is low error on the examples it trained on and high error on examples it has never seen. That’s why you always hide some data from the model. More on that next lesson.
Key takeaways
- Overfitting = memorising the training examples, noise and all, instead of learning the pattern.
- Its signature: training error near zero, error on new examples high.
- Underfitting is the opposite: too simple to catch the real pattern, so it’s bad everywhere.
- The right flexibility sits in between, and only data the model never trained on can show you where.
Check your understanding
Go deeper (more technical)
The simulation fits polynomials: curves of the form a + bx + cx² + …. The flexibility slider sets the highest power (the “degree”). With 10 training points, a degree-9 polynomial has 10 numbers to tune and can pass through every point exactly: the purest form of memorising.
Practitioners fight overfitting with more data, simpler models and regularisation: tricks that make large weights cost extra (weight decay) or randomly switch off neurons during training (dropout), so the network can’t lean on any one quirk. Another is early stopping: watch the error on held-out data and stop training when it starts to rise.
Giant modern networks complicate the story: some keep improving on new data even after they fit their training set perfectly, a puzzle known as “double descent”. The basic rule still holds, though: judge a model on data it has never seen.