Why Some AI Models Are Faster and Cheaper Than Others
Understand the levers that make some models much faster and cheaper than others, and how to pick one.
Open any AI provider’s price list and you’ll see the same family of models at wildly different prices: a “mini” model might cost a fiftieth of the flagship per token and answer three times faster. After this track, you can explain exactly where that difference comes from.
Remember the golden rule from Track 3: each decode step must read the weights the model uses, so speed and cost follow the number of bytes read per token. Four levers shrink it:
- Fewer parameters. An 8B model reads a ninth of the bytes of a 70B one. To keep quality up, small models are often trained by imitating a big one, called distillation.
- Fewer bits per parameter. Quantization (Track 2) to 8 or 4 bits halves or quarters the bytes.
- Fewer active parameters. A mixture-of-experts model (Track 2) stores many parameters but uses only a fraction per token: big-model knowledge at small-model speed, if you have the memory.
- Fewer tokens. A reasoning model may write thousands of hidden thinking tokens before answering. Each one costs as much as a visible one.
On top of the model itself come the serving tricks you’ve just learned, newer GPUs, and sheer scale: a provider with huge, steady traffic keeps its batches full and its GPUs busy.
None of this is free. Smaller, more compressed models know less and make more mistakes on hard problems, and skipping the thinking costs accuracy on tricky ones. So the real skill is matching the model to the job: a small, cheap model for millions of simple questions, a big or reasoning model for the hard ones. That choice is exactly where Track 5 picks up, when you start building applications of your own.
Key takeaways
- Speed and cost follow the bytes read per generated token, and the number of tokens generated.
- Levers: fewer parameters (often via distillation), fewer bits, fewer active parameters (MoE), fewer tokens.
- Serving tricks, hardware and scale lower costs further.
- Cheaper models give up some quality, so choose by the job, not by the leaderboard alone.
Check your understanding
Go deeper (more technical)
Distillation was popularised by Hinton and colleagues (2015): the student learns from the teacher’s full probability distribution over answers, not just the right answer, which carries extra information. Many small open models today are trained partly on text written by larger models.
Specialised chips (for example Groq’s LPUs or Cerebras’ wafer-scale engines) keep weights in very fast on-chip memory to reach hundreds or thousands of tokens per second per user, a different trade-off of cost, capacity and speed.
The simulation’s models are generic and its numbers illustrative.