Why AI Is So Computationally Expensive
See why AI needs an enormous amount of very simple maths, and how to estimate exactly how much.
Track 1 took the magic out of neural networks: each neuron multiplies its inputs by its weights, adds them up and adds a bias. Track 2 showed that an LLM is that same recipe stacked into a transformer. So when ChatGPT writes a word, what is the computer actually doing? Overwhelmingly: multiplying and adding.
Each multiply or add is one FLOP (a floating-point operation), and the rule of thumb is beautifully simple: one generated token costs about 2 FLOPs per parameter. A 500-word answer from that 8B model is therefore around 10 trillion operations. GPT-3, with 175 billion parameters, needs about 350 billion operations per word.
Training is far hungrier. Every training token goes forward through the network and then backward to work out the nudges, about 6 FLOPs per parameter in total, and it’s repeated over trillions of tokens. Training GPT-3 took roughly 3 × 10²³ operations. Written out, that’s a 3 followed by 23 zeros.
Here’s the good news hiding in those numbers. None of these operations is hard: no clever logic, no decisions, just multiply and add, over and over. So the question “what’s the best chip for AI?” turns into a much simpler one: which chip can do the most simple sums per second? That question is the thread running through this whole track.
Key takeaways
- Neural networks do almost nothing except multiply and add, but a staggering number of times.
- One generated token costs about 2 operations (FLOPs) per parameter: one multiply and one add.
- Training costs about 6 FLOPs per parameter per training token, repeated over trillions of tokens.
- Because every operation is simple, the best AI chip is the one that does the most simple sums at once.
Check your understanding
Go deeper (more technical)
Inference rule of thumb: generating one token costs about 2 · N FLOPs, where N is the number of parameters. Attention adds a little more for long contexts, roughly proportional to the context length.
Training rule of thumb: about 6 · N · D FLOPs, where D is the number of training tokens: 2 for the forward pass and about 4 for the backward pass. For GPT-3 (N = 175 billion, D ≈ 300 billion tokens) that is roughly 3.1 × 10²³ FLOPs. Even at an ideal 1,000 trillion FLOP/s on one GPU, that is about 10 years of nonstop work, which is why it was trained on thousands of GPUs at once. The last lesson of this track returns to that calculation.
Figures are approximate and based on published model sizes.