What Happens After You Press Send?
Follow one message from your keyboard to a finished answer, and see where the time actually goes.
Track 2 opened up the model. Track 3 opened up the chip. This track asks the question in between: what does it take to run that model for millions of people at once, quickly and without going bankrupt? Let’s start by following a single message.
You press send. Your message flies across the internet to a data centre, usually tens of milliseconds away. A gateway checks who you are, whether you’ve hit your usage limit and, often, whether the request is allowed by safety filters. Then it joins a queue, because thousands of other people pressed send in the same second and every GPU slot might be taken.
When a slot frees up, the text is tokenized, wrapped in its chat template, and the model gets to work in two very different phases you’ll meet properly in the next lesson. First, prefill: the whole prompt is read in one big parallel pass. Then decode: the answer is written one token at a time, and each token is sent back to your screen the moment it exists.
Each stop costs time, and which one dominates depends on the situation. On a quiet day with a short question, the decode phase usually takes most of the time. When the service is overloaded, you may sit in the queue for seconds before anything happens. Paste in a 300-page document and prefill takes a noticeable bite too.
Serving engineers think about exactly these trade-offs, every day: how to keep the queue short, make the first word appear fast, keep the words flowing, and squeeze as many users as possible onto each very expensive GPU. The rest of this track is a tour of their toolbox.
Key takeaways
- A message passes through the internet, a gateway, a queue, then the GPU, and the answer streams back.
- On the GPU, the prompt is read in one parallel pass (prefill), then the answer is written token by token (decode).
- Which stage dominates depends on traffic, prompt length and answer length.
- Serving is about keeping all of it fast while fitting as many users as possible on each GPU.
Check your understanding
Go deeper (more technical)
Real systems also put a load balancer in front of many model replicas, choose a replica that already holds the start of your conversation in memory (more on prefix caching later), and apply rate limits per user or per API key, often counted in tokens per minute.
The streamed tokens usually travel back over a long-lived HTTP connection using server-sent events, one small message per token or group of tokens. That’s why API responses can be read as a stream as soon as they start.
The simulation’s stage timings are illustrative.