Spark LampHow Does AI Serve Millions of People?
What happens after you press send: prefill and decode, batching, the KV cache, splitting models over GPUs, serving engines, and what each answer costs.
Modules
Each module explains one idea in plain words, lets you try it, then checks what stuck.
- 01
What Happens After You Press Send?
Follow one message from your keyboard to a finished answer, and see where the time actually goes.
- 02
Prefill vs Decode: The Two Phases of LLM Inference
Understand why reading the prompt and writing the answer stress the GPU in completely different ways.
- 03
Why AI Responds Quickly but Generates Slowly
Learn the two numbers that describe how fast a chatbot feels, and why streaming changes everything.
- 04
Latency vs Throughput: Speed Isn't Just One Number
See why a GPU can’t be both fastest for one user and most productive overall, and how services choose.
- 05
Batching: How One GPU Handles Multiple Users
See how one GPU serves many users with a single pass, and the flaw in the simplest way of doing it.
- 06
Continuous Batching: Keeping GPUs Busy
See how serving engines refill every free seat on every step, and why that multiplies throughput.
- 07
KV Cache: Why AI Doesn't Recalculate Everything
Learn the trick that makes long answers fast, and why it turns memory into the scarcest resource on the GPU.
- 08
PagedAttention: Managing AI Memory Efficiently
See how borrowing an old operating-system trick lets a GPU fit far more conversations into the same memory.
- 09
Speculative Decoding: Can a Small Model Speed Up a Big One?
See how a small, fast model can make a big model write faster, without changing a single word it writes.
- 10
When One GPU Isn't Enough
Learn the ways a model is spread across several GPUs, and what each one buys you.
- 11
Inside an LLM Serving Engine
Put the whole toolbox together and see how a real serving engine is organised.
- 12
How Much Does One AI Response Actually Cost?
Work out what one AI answer really costs to produce, and why idle GPUs and long answers are expensive.
- 13
Why Some AI Models Are Faster and Cheaper Than Others
Understand the levers that make some models much faster and cheaper than others, and how to pick one.