Streaming: Displaying Answers as They're Generated
See how an app receives and displays an answer piece by piece, and the small problems that creates.
Track 4 explained why streaming makes AI feel fast: the model writes one token at a time, so showing each token as it arrives lets people start reading straight away. This lesson looks at the same thing from the app builder’s side. What actually arrives, and what does the app have to do with it?
With streaming switched on in the request, the API doesn’t send one response at the end. It keeps the connection open and sends a series of small events, each carrying a fresh piece of text, often called a delta. A common way to do this on the web is “server-sent events”: lines that start with data:, arriving one after another. The last few events carry the finishing details: why the model stopped, and how many tokens were used.
The app’s job is to stitch the pieces together as they land, and that brings a few practical wrinkles:
- Stopping. If the user presses stop, the app closes the connection. Generation ends, and the tokens never written are never paid for.
- Half-finished formatting. Text like
**boldor half a table can flicker oddly while it’s incomplete, so apps render markdown carefully as it grows. - Half-finished data. A JSON object can’t be parsed until its closing bracket arrives. Apps either wait for the end or use a tolerant parser that reads partial data.
- Interruptions. Connections drop, so apps keep what arrived and can retry.
When is streaming not worth it? When nobody is watching: a background job that sorts a thousand emails gains nothing from seeing each label being typed. Streaming is for people; programs can wait for the full answer.
Key takeaways
- A streamed response arrives as a series of small events, each carrying a few new characters.
- The app appends each piece as it lands; the final events say why it stopped and how many tokens were used.
- Stopping a stream ends generation, so unwritten tokens aren’t paid for.
- Partial markdown and partial JSON need care; background jobs often don’t need streaming at all.
Check your understanding
Go deeper (more technical)
Server-sent events (SSE) are a simple web standard: the server keeps an HTTP response open and writes lines beginning with data:. Some APIs instead use WebSockets, which allow messages in both directions, useful for real-time voice (Track 8).
Streaming also helps with timeouts: a long answer that would take 60 seconds can trip network time limits if nothing is sent until the end, while a stream keeps the connection visibly alive. Some providers report usage only in the final event, so apps that stop early must count tokens themselves.
The event format in the simulation is a generic example.