Context Windows: How Much Can an LLM Read at Once?
Understand what a context window is, what fills it, and what happens when a conversation outgrows it.
Here’s something that surprises a lot of people: a chatbot has no memory between your messages. Every time you press Enter, the app sends the model the entire conversation so far (system instructions, every earlier message, your new one) and the model reads it all from scratch to write its reply.
The most text it can take in at once is its context window, measured in tokens. Everything has to fit: the hidden system prompt, the conversation history, any documents you pasted in, and the answer it’s about to write.
What happens when a long chat outgrows the window? The app has to drop something, usually the oldest messages. From then on, the model doesn’t vaguely remember them; as far as it’s concerned, they never happened. That’s how a bot that knew your name this morning can ask for it again this afternoon. Some apps soften this by summarising old messages or saving notes, but those are workarounds, not a bigger window.
Windows have grown at a dizzying pace: 2,048 tokens for GPT-3 in 2020 (a few pages), 128,000 by late 2023 (a whole novel), a million or more in some models from 2024 (a shelf of novels).
So why not make them infinite? Attention compares every token with every other, so its work grows with the square of the length: double the text, four times the comparisons. While writing, the model must also keep keys and values for every token in memory (the KV cache, Track 4), which for long contexts can outgrow the model itself. And having room for a text is not the same as using all of it well, which is the next lesson.
Key takeaways
- The model doesn’t remember chats: the whole conversation is re-sent with every message.
- The context window is the most tokens it can read at once: system prompt, history, documents and answer together.
- When a chat outgrows it, the oldest messages are dropped and are effectively never seen again.
- Longer windows cost more: attention work grows with length squared, and the KV cache grows with every token.
Check your understanding
Go deeper (more technical)
KV cache per token = 2 (keys and values) × layers × KV heads × head size × bytes. For Llama 3 8B in 16-bit: 2 × 32 × 8 × 128 × 2 bytes = 128 KiB per token, so a full 128K-token context needs about 17 GB, more than the 16 GB of weights.
Models are usually pre-trained on shorter texts, then extended with extra training on long documents, often combined with stretching the rotary position angles from the word-order lesson. Some models use sliding-window attention, where each token only looks at a fixed number of recent tokens in certain layers, to keep costs down.