Can We Make an AI Model Smaller Without Ruining It?
See how models are shrunk to fewer bits per weight, and the trick that keeps them accurate.
A trained model is a giant pile of numbers, usually stored in a 16-bit format: 2 bytes per weight. A 70-billion-parameter model is therefore about 140 GB, more than any single consumer GPU can hold. Can we store each weight in fewer bits without breaking the model?
That’s quantization: rounding every weight to one of a small set of allowed values. 8 bits allow 256 different values; 4 bits only 16. Picture a ladder with 16 rungs, where every weight snaps to its nearest rung.
Surprisingly, models put up with this well. Neural networks are full of redundancy, and small rounding errors spread over millions of weights mostly cancel out. 8-bit is usually close to lossless; 4-bit costs a little quality but cuts memory to a quarter, so that 140 GB model shrinks to about 35 GB. Generation gets faster too, because each token needs fewer bytes moved out of memory (Track 3 explains why moving bytes, not doing maths, is usually the bottleneck).
The rungs are spread evenly between minus and plus the biggest weight. And there’s the catch: LLMs contain rare outlier weights far bigger than the rest. One giant weight stretches the ladder so far that the rungs end up wide apart, and thousands of ordinary small weights all get rounded to the same rung near zero, wiping out their information.
The fix is simple and very effective: don’t use one ladder for a whole layer. Give every small group of weights, say 32 or 128 of them, its own scale. An outlier then only stretches its own group’s ladder, and everyone else keeps closely spaced rungs. The extra scales cost only a fraction of a bit per weight, and this is the trick behind the popular 4-bit formats that let capable models run on a laptop.
Key takeaways
- Quantization rounds each weight to a small set of allowed values.
- Fewer bits = smaller and faster, but bigger rounding errors.
- Rare outlier weights stretch the scale and hurt everything else.
- A separate scale for each small group of weights keeps 4-bit models accurate.
Check your understanding
Go deeper (more technical)
This simulation uses symmetric “absmax” rounding: scale = max|w| / (2^(bits−1) − 1). Real methods are smarter: GPTQ adjusts the remaining weights to compensate for each rounding error; AWQ protects the weights that matter most for the activations; common group sizes are 32–128 weights.
Outliers are a well-documented feature of LLMs (LLM.int8(), 2022), especially in activations. Formats like FP8, and newer 4-bit floating-point formats with per-block scales (MXFP4, NVFP4), are supported directly by recent GPUs. 8-bit is usually nearly lossless; 4-bit costs a little quality; 3- and 2-bit cost noticeably more.