Continuous Batching: Keeping GPUs Busy
See how serving engines refill every free seat on every step, and why that multiplies throughput.
Static batching’s problem was its timetable. Groups were formed only between batches, so finished requests left empty seats and new arrivals waited at the door. The fix sounds almost too simple: stop waiting.
With continuous batching, the batch is rebuilt at every single decode step. The moment a request writes its final token, its seat goes to the next request in the queue, on the very next step. Long answers and short answers flow through side by side, and the GPU almost never runs a step for an empty seat.
Think of a busy restaurant. Static batching seats a whole table of eight and won’t seat anyone else at it until the slowest diner has finished dessert. Continuous batching clears a chair the second someone leaves and seats the next guest straight away. Same restaurant, same kitchen, far more meals per evening.
The new arrival still needs its prompt read before it can start writing. Engines handle that by mixing work: one step can include decoding for most requests plus a slice of a newcomer’s prefill. Splitting long prompts into such slices (“chunked prefill”) stops a huge prompt from freezing everyone else’s stream of tokens.
The idea was introduced in 2022 in a system called Orca, under the name iteration-level scheduling. It made such a difference, often several times more throughput on real chat traffic, that every major serving engine now does it. If the last lesson was “share the bus”, this one is “never let the bus run with empty seats”.
Key takeaways
- Continuous batching rebuilds the batch at every decode step.
- A finished request’s seat goes to the next waiting request immediately.
- Newcomers’ prompts are mixed into ongoing steps, often in chunks, so nobody’s stream stalls.
- It raises throughput and cuts waiting time, which is why every major engine uses it.
Check your understanding
Go deeper (more technical)
Orca (Yu and colleagues, OSDI 2022) introduced iteration-level scheduling. It is now standard in vLLM, SGLang, Hugging Face TGI and NVIDIA TensorRT-LLM (which calls it in-flight batching).
The scheduler’s job at each step is a small puzzle: which waiting requests fit in memory, how many prompt tokens to prefill alongside the decodes, and whether to pause (“preempt”) a request when memory runs short. The next two lessons are about that memory.
The simulation’s requests and timings are a toy example.