CPUs vs GPUs: A Few Experts vs an Army of Workers
Understand the different design ideas behind a CPU and a GPU, and when each one wins.
Your laptop already has a powerful chip, the CPU. Why not run AI on that? Because CPUs and GPUs are built on opposite philosophies.
A CPU is a small team of brilliant experts. It has a handful of powerful cores, usually 8 to 64, and spends much of its chip on making each one fast and clever: guessing which way an “if” will go, shuffling instructions into a faster order, keeping big caches of data close by. That’s perfect for the messy, varied work your computer does all day: a browser, a spreadsheet, an operating system juggling a hundred small tasks.
A GPU is an army. It has thousands of simple cores that are individually slower and much less clever, but all work in parallel. GPUs were invented to colour the millions of pixels on a screen, sixty times a second: a mountain of tiny, identical, independent jobs. Ask a GPU to do one thing, and it’s sluggish. Ask it to do the same thing a million times, and nothing comes close.
A neural network turns out to look exactly like a screen full of pixels: millions of multiplications that don’t depend on each other. Researchers had noticed this for years, and in 2012 a network called AlexNet, trained on two gaming graphics cards, crushed the ImageNet competition (Track 1’s “Why did AI suddenly get so good?”). AI has run on GPUs ever since.
The army isn’t always the answer, though. Getting it started costs time, and if every step needs the result of the one before, extra soldiers just stand around. The race on the right lets you find both cases.
Key takeaways
- CPU: a few fast, flexible cores. Great for varied work and step-by-step logic.
- GPU: thousands of simple cores. Great when the same small job is repeated on lots of data.
- Neural network maths is exactly that kind of job, which is why AI runs on GPUs.
- GPUs lose on tiny jobs (start-up cost) and on chains where each step waits for the last.
Check your understanding
Go deeper (more technical)
A modern server CPU spends much of its chip area on large caches, branch prediction and out-of-order execution, all to make a single thread fast (low latency). A GPU spends its area on many simple arithmetic units and hides delays by switching between thousands of threads (high throughput).
For example, an NVIDIA H100 has 132 streaming multiprocessors with a total of 16,896 FP32 cores plus 528 tensor cores. A fast desktop CPU might reach around 1–2 trillion FP32 operations per second with vector instructions; the H100 reaches about 67 trillion in plain FP32 and about 989 trillion with tensor cores in BF16.
The race in the simulation is a toy model with made-up timings chosen to show the trade-off, not a benchmark.