Why Bigger Context Windows Don't Always Mean Better Memory
See why a model that can read a huge input doesn’t necessarily use all of it well.
A million-token window sounds like perfect memory: paste in a whole codebase or a stack of contracts and the model “knows” all of it. In practice, being able to read a long text and being able to use all of it are two different things.
In 2023, researchers from Stanford and elsewhere ran a neat test. They gave models a question plus a pile of documents, only one of which held the answer, and moved that document around. Models did best when the key document was at the very start or the very end, and markedly worse when it sat in the middle. The paper was called “Lost in the Middle”, and its U-shaped curve has shown up in many variations since.
Why would that happen? A few plausible reasons. Attention has to spread over every token, so in a huge context each one gets a thinner slice. Models see far fewer very long documents in training than short ones. And the start of a text (titles, instructions) and its end (closest to the question) usually matter most in training data, so models learn to lean on both.
Newer models have improved a lot. They ace simple “needle in a haystack” tests, where one odd sentence is hidden in a long document. But harder tasks over long inputs, such as connecting several facts scattered far apart or noticing what’s missing, still tend to degrade as the input grows. A model’s advertised window is the most it can read; its effective window, where it reasons well, is often smaller.
The practical lessons: put the most important material and your question near the start or the end, cut what isn’t needed, and for big collections of documents, retrieve only the relevant pieces instead of pasting everything. That last idea, RAG, gets a whole course of its own in Track 6.
Key takeaways
- Fitting into the window doesn’t guarantee being used well.
- Models tend to use information at the start and end of long inputs better than in the middle: “lost in the middle”.
- Simple needle-finding tests are mostly solved; harder reasoning over very long inputs still degrades.
- Put key information and questions at the edges, trim the input, or retrieve only what’s relevant.
Check your understanding
Go deeper (more technical)
“Lost in the Middle” is Liu and colleagues (2023). The needle-in-a-haystack test was popularised by Greg Kamradt in 2023. Benchmarks such as RULER (2024) test harder long-context skills, like tracking variables or counting across a document, and found that many models perform well only on a fraction of their advertised length.
The simulation’s numbers are illustrative, shaped like the published curves; real results depend heavily on the model and the task. Researchers have also found that models dump a surprising share of attention on the first few tokens (“attention sinks”), one hint at why position matters so much.