Latency vs Throughput: Speed Isn't Just One Number
See why a GPU can’t be both fastest for one user and most productive overall, and how services choose.
“How fast is it?” sounds like a simple question. For an AI service it has two very different answers.
Latency is how fast it is for you: how long until your answer arrives, or how many tokens per second you see. Throughput is how much work the whole GPU gets done: total tokens per second, summed over everyone it serves. A sports car has superb latency for one passenger. A bus is slower for each person, but moves far more people per hour.
On a GPU, the dial between the two is the batch size: how many users share each decode step. Thanks to the memory wall, the first steps up are almost free. A decode step with 32 users still mostly waits for the weights to stream in, so it takes barely longer than a step with one user, while producing 32 tokens instead of one. Total throughput shoots up and each user hardly notices.
Push further and the costs arrive. Every user brings their own KV cache, which must also be read at every step, and eventually the maths units fill up too. Steps get slower, so each user’s tokens per second fall, while total throughput flattens out. Somewhere along that curve, a service has to choose.
Real services choose with a promise called a service-level objective: for example, “first token within a second, and at least 30 tokens per second for everyone”. The job is then to pick the biggest batch that still keeps the promise, because every extra user per GPU makes each answer cheaper. A free tier might accept slower answers to fit more users; a premium tier might do the opposite.
Key takeaways
- Latency = how fast for one user. Throughput = total work for everyone.
- Bigger batches raise throughput; at first almost for free, because decode is memory-bound.
- Large batches slow every user down, as KV-cache reads and the maths start to add up.
- Services set targets (TTFT, tokens per second per user) and pack in as many users as those allow.
Check your understanding
Go deeper (more technical)
The useful measure is often goodput: the throughput of requests that actually met their latency targets. Pushing the batch too high can raise raw throughput while goodput falls, because more requests miss their targets.
Latency also includes queueing. When a service runs close to full capacity, queues grow sharply (a classic result of queueing theory), so operators keep some headroom and add replicas before demand peaks.
The simulation models an 8B model on one H100 with a 2,000-token context per user (128 KiB of KV cache per token). Its numbers are estimates, not benchmarks.