Why Did AI Suddenly Get So Good?
Understand the three ingredients that came together around 2012 and set off the AI boom.
Here’s the odd thing. Almost everything in this course so far is old. The artificial neuron dates from 1943. Backpropagation caught on in 1986. By 1989, a neural network was reading handwritten zip codes from US mail.
So why did AI seem to explode overnight? Because a recipe is only as good as its scarcest ingredient, and for decades, not all of them were there:
- Data. Neural networks are ravenous. The internet finally fed them: in 2009, ImageNet offered over a million labelled photos to train on.
- Compute. Training is mountains of multiply-and-add. Graphics cards built for video games turned out to do exactly that, thousands of sums at a time.
- Ideas. The basics were old, but deep networks needed new tricks to train well, like ReLU and dropout.
In 2012 they met. A network called AlexNet, trained on two gaming graphics cards, cut the error in the ImageNet competition from about 26% to 15%. Within a few years nearly every lab had switched to deep learning.
Then 2017 brought the transformer, and the discovery that the same recipe kept improving as all three ingredients scaled up. That road leads straight to ChatGPT.
Key takeaways
- The core ideas (neurons, backpropagation, even image networks) were decades old by 2012.
- What changed: big labelled datasets, GPUs that made training fast, and tricks that made deep networks trainable.
- AlexNet (2012) proved it by slashing the ImageNet error rate, and the deep learning boom began.
- Progress since then has come from scaling all three ingredients up together.
Check your understanding
Go deeper (more technical)
The 1989 zip-code reader came from Yann LeCun and colleagues at Bell Labs; its descendants were reading a large share of US cheques by the late 1990s. Then neural networks fell out of fashion for a decade, as other methods matched them with less fuss.
ImageNet, led by Fei-Fei Li, held about 14 million labelled images; its yearly competition used 1.2 million of them across 1,000 categories. AlexNet (Krizhevsky, Sutskever and Hinton) trained for five to six days on two NVIDIA GTX 580 cards. Its top-5 error (the right answer missing from its five best guesses) was 15.3%, against 26.2% for the runner-up.
Since then, the computing power used to train the largest models has grown roughly four- to fivefold a year. Track 3 explains why GPUs are so good at this, and why memory, not arithmetic, is often the real limit.