Why AI Responds Quickly but Generates Slowly

Ask a chatbot something and two separate things happen: a short pause, then words that flow steadily onto the screen. Serving engineers measure these separately, because they have different causes and different fixes.

Time to first token (TTFT) is the pause: queueing, plus prefill of your whole prompt, plus the network. It’s what makes an app feel responsive or sluggish. It grows with long prompts, busy services and, for reasoning models, with all the thinking that happens before the visible answer.

Time per output token (TPOT, also called inter-token latency) is the rhythm after that: how long each decode step takes. Its inverse is the familiar “tokens per second”. It depends on the model’s size, the GPU’s memory bandwidth, and how many other users share each step.

Here’s the trick that makes chatbots feel fast: streaming. Instead of waiting for the full answer, the app shows each token the moment it exists. A 400-word answer at 50 tokens a second takes about ten seconds to finish, but you start reading after half a second, and you can’t read faster than about four or five words a second anyway. As long as the model writes faster than you read, it feels instant.

That gives serving teams a useful target. TTFT should be short, ideally well under a second. TPOT only needs to stay comfortably faster than reading speed. Any spare speed beyond that can be traded for something more valuable, such as serving more people per GPU, which is exactly the trade the next lesson explores.