Training vs Inference: Learning vs Using
Know the two jobs of every model, training and inference, and why both need huge amounts of math.
A model has two very different lives. training is when it learns: it sees an example, makes a guess (the forward pass), compares it with the right answer, then goes back through the network to nudge every weight (the backward pass). This repeats over millions or trillions of examples.
inference is when it is used: you ask, it answers. Only the forward pass runs, and the weights are frozen: the model does not learn from your questions. Every answer from ChatGPT is inference.
Both jobs are mountains of multiplications and additions: for every answer, roughly every weight is used. Train once (huge cost, done by the model maker), then run inference millions of times (a smaller cost per answer, but paid for every single user). That is why AI needs very special hardware, which is exactly where Track 3 picks up.
Key takeaways
- Training = forward pass + compare + backward pass, repeated over huge amounts of data. The weights change.
- Inference = forward pass only. The weights are frozen.
- One training step costs about 3 times the math of one answer, and training repeats it over billions of examples.
- Every weight must be stored in memory and used for every answer: the reason AI needs special hardware (Track 3).
Check your understanding
Go deeper (more technical)
Rules of thumb used by AI engineers: a forward pass costs about 2 · N operations for a model with N parameters, and a training step about 6 · N per example (2 forward, about 4 backward). Training also needs much more memory: besides the weights, it keeps the gradients and extra bookkeeping for every weight.
Training a large model is usually done once, on thousands of chips for weeks or months. Inference runs for the rest of the model's life, for every user, so over time the total cost of inference often ends up larger than the cost of training.
Llama 3 8B figures: about 8 billion parameters trained on about 15 trillion tokens (Meta, 2024).