How Much GPU Memory Does an AI Model Need?
Put it all together: estimate the memory, the number of GPUs and the speed needed to run a real model.
Time to cash in everything you’ve learned on the question every AI team actually asks: what hardware do we need to run this model? Two back-of-the-envelope formulas get you surprisingly far.
1. Will it fit? For inference, the big cost is storing the weights: parameters × bytes per parameter. An 8-billion-parameter model in BF16 is 8 × 2 = 16 GB. On top of that comes working memory, mostly the KV cache from Track 2, which grows with the conversation length and the number of users. A flat 20% extra is a decent first guess for a few users.
When a model doesn’t fit, there are two escape routes. Split it across several GPUs, which then pass results between each other over fast links (Track 4 explains how). Or shrink it with quantization: the same 70B model is 140 GB in BF16 but about 35 GB in 4-bit, small enough for one 80 GB card.
2. How fast will it write? Remember the memory wall: for one user, every new token requires reading every weight once. So the speed limit is simply memory bandwidth ÷ model size. An H100 streams 3.35 TB/s; divide by 16 GB and you get a ceiling of about 200 tokens a second. Halve the bytes per weight and the ceiling doubles. That’s why quantized models aren’t just smaller, they’re faster.
Real systems land below these ceilings, and serving many users changes the picture (Track 4 again). But these two sums will tell you within minutes whether a plan is sensible or fantasy.
Key takeaways
- Memory for running a model ≈ parameters × bytes per parameter, plus room for the KV cache and working memory.
- Single-user speed ≈ memory bandwidth ÷ model size, because each token reads every weight.
- Too big? Split the model across GPUs, or quantize it.
- Quantization makes a model both smaller and faster, at a small cost in accuracy.
Check your understanding
Go deeper (more technical)
Inference memory = weights + KV cache + activations + framework overhead. The KV cache grows with context length and the number of concurrent users, and can rival the weights in size (more in the serving track). The flat 20% in the calculator is a simplification.
The bandwidth formula gives an upper bound for one user. Real systems typically reach a good fraction of it; with many users, the work becomes more compute-bound and total throughput rises even as each user’s speed falls.
GPU specs (approximate): RTX 4090 24 GB / 1.0 TB/s, A100 80 GB / 2.0 TB/s, H100 80 GB / 3.35 TB/s, H200 141 GB / 4.8 TB/s, B200 192 GB / 8 TB/s.