Inside an LLM Serving Engine
Put the whole toolbox together and see how a real serving engine is organised.
You’ve now met every major trick in the serving toolbox. In practice nobody wires them up by hand. They come bundled in a serving engine: software such as vLLM, SGLang, NVIDIA’s TensorRT-LLM or Hugging Face’s TGI that takes a model file and turns it into a fast, shared service.
Inside, most engines have the same four parts:
- API server. Accepts requests, tokenizes them, and streams tokens back as they’re made.
- Scheduler. The brain. Before every step it decides who runs: which requests continue decoding, which newcomers get a slice of prefill, who waits, and who gets paused if memory runs short. Continuous batching lives here.
- KV cache manager. Hands out memory pages, reuses shared prefixes, and frees pages when requests end. PagedAttention lives here.
- Model runner. Executes each step on the GPUs, split across them if needed, using heavily optimised kernels and often quantized weights and speculative decoding.
The engine runs a tight loop, many times a second: schedule, run one step, stream out the new tokens, free what’s finished, repeat. Each step can serve dozens or hundreds of conversations.
The tricks multiply rather than add. Batching raises throughput; continuous batching keeps the batch full; paging lets the batch grow; prefix caching skips repeated prefills; quantization and speculative decoding speed up every step. Together, the same GPU can serve many times more people than a naive server would, and that difference is a big part of why AI chat is affordable at all.
Key takeaways
- A serving engine bundles the serving tricks into one program: vLLM, SGLang, TensorRT-LLM, TGI and others.
- Four parts: API server, scheduler, KV cache manager and model runner.
- The engine loops: schedule, run one step, stream new tokens, free finished requests.
- The tricks multiply, turning the same GPU into a far more productive server.
Check your understanding
Go deeper (more technical)
vLLM came from UC Berkeley (2023) with PagedAttention; SGLang (2024) added RadixAttention prefix sharing and a fast scheduler; TensorRT-LLM compiles models into highly optimised GPU kernels for NVIDIA hardware. All expose an OpenAI-compatible API, so apps can switch between them easily.
Engines also handle structured output (forcing valid JSON by masking impossible tokens), LoRA adapters swapped per request (Track 2), and preemption: when memory runs short, a request’s cache can be dropped and recomputed later, or moved to CPU memory.
The simulation’s multipliers are illustrative.