Why Training Needs Far More Hardware Than Running a Model
See why training a model takes vastly more memory and compute than running it, and how clusters make it possible.
Running a 70B model takes one or two GPUs. Training one took Meta thousands. Where does that gap come from? Two places: memory and time.
Memory. Running a model needs the weights, about 2 bytes per parameter. Training needs a lot more company for each weight:
The gradients (how to nudge each weight, from backpropagation in Track 1), an FP32 master copy so tiny updates aren’t rounded away, and the optimiser’s two running averages per weight that make training stable. That’s about 16 bytes per parameter, eight times the cost of running, before even counting the activations: every layer’s intermediate results, kept from the forward pass for the backward pass.
For a 70B model that’s over a terabyte. No GPU comes close, so the bookkeeping is sharded: split across many GPUs, each holding its own slice and swapping pieces with the others as needed. Activations get the opposite trick, checkpointing: throw most of them away and recompute them during the backward pass, trading a bit of extra maths for a lot of memory.
Time. From the first lesson: training costs about 6 FLOPs per parameter per token. Llama 3 405B saw about 15 trillion tokens: roughly 3.6 × 10²⁵ operations. A single H100 achieving 40% of its peak would need thousands of years. Meta used up to 16,384 H100s, which brings it down to a couple of months.
And that’s the full circle of this track: AI is simple maths, done in staggering amounts, on hardware whose real limit is moving data. Track 4 takes the trained model and asks the next question: how do you serve it to millions of people at once?
Key takeaways
- Running a model needs about 2 bytes per parameter; training needs about 16, plus activations.
- The extras: gradients, an FP32 master copy and the optimiser’s state for every weight.
- Sharding spreads that memory over many GPUs; checkpointing trades extra maths for less activation memory.
- Training compute ≈ 6 × parameters × tokens, which is why big models need thousands of GPUs for months.
Check your understanding
Go deeper (more technical)
Training memory with Adam in mixed precision: 2 bytes (BF16 weights) + 2 (gradients) + 4 (FP32 master weights) + 8 (two FP32 optimiser states) = 16 bytes per parameter, plus activations, which grow with batch size and sequence length. ZeRO (Microsoft, 2019) and PyTorch’s FSDP shard these states across GPUs.
Big runs combine several kinds of parallelism: data parallel (each GPU group sees different data), tensor parallel (each matrix split across GPUs), pipeline parallel (different layers on different GPUs) and sometimes expert parallel for mixture-of-experts models. Meta reported training Llama 3 405B on up to 16,384 H100 GPUs using about 3.8 × 10²⁵ FLOPs, with around 38–43% of peak utilisation.
The simulation assumes 40% of an H100’s 989 TFLOP/s BF16 peak; its activation figures are illustrative.