How AI Generates an Answer, One Token at a Time
Follow one question all the way from your keyboard to a finished answer, pass by pass.
Let’s follow one question all the way through. You type “What is the capital of France?” and press Enter.
1. Wrap it up. Your message isn’t sent alone. The app wraps it in a chat template: a hidden system message (“You are a helpful assistant…”), markers saying who is speaking, and a final marker meaning “the assistant’s turn starts here”. Those markers are special tokens the model saw countless times during fine-tuning; they’re how it knows to answer rather than, say, continue your question.
2. Read everything. The whole thing is tokenized and pushed through the transformer in one go. All the prompt tokens are processed in parallel, so even a long prompt needs only one pass. Out comes a probability for every token in the vocabulary.
3. Pick and append. One token is chosen (“The”), stuck on the end, and the model runs again with one more token. Every pass yields exactly one new token. That’s why answers appear word by word on screen: the app shows each token the moment it exists.
4. Know when to stop. The model ends its answer the same way it writes it: by predicting a special end-of-turn token. Apps also set a maximum length as a safety net; hit it, and the answer is simply cut off mid-sentence.
Notice the asymmetry. Reading your prompt is quick because it’s parallel. Writing is slow because it’s strictly one token after another: a 500-token answer needs 500 passes through the entire model. That imbalance shapes almost everything about how AI is served, which Track 4 takes apart.
Key takeaways
- Your message is wrapped in a chat template with special tokens before the model sees it.
- The prompt is read in one parallel pass; after that, each pass adds exactly one token.
- The model stops by generating an end-of-turn token, or when the app’s length limit cuts it off.
- Reading is fast and parallel; writing is slow and sequential, which is why long answers take time.
Check your understanding
Go deeper (more technical)
Engineers call the first pass prefill and the token-by-token passes decode. Thanks to the KV cache (Track 4), decode passes don’t recompute the earlier tokens; they only add the new one, but they must still read the model’s full weights every time.
Every model family has its own template: ChatML uses <|im_start|> and <|im_end|>; Llama 3 uses <|start_header_id|> and <|eot_id|>. Feeding a model the wrong template noticeably degrades its answers. APIs also accept stop sequences, extra strings that end generation when they appear.