Chunking: Why Document Size Matters
See why documents are split into chunks, and how the size and placement of the splits decide what can be found.
Clean text is still too big to use whole. A 200-page manual has one embedding that tries to summarise everything, so it matches no specific question well. And putting all 200 pages into the prompt would be slow and expensive, and would bury the answer. So documents are cut into smaller pieces before embedding: chunking. Each chunk gets its own vector and becomes a separate search result.
The size is a balancing act.
- Too small and ideas get cut in half. “Bikes can be returned within 30 days.” lands in one chunk, “This does not apply to custom-built bikes.” in the next. Retrieve only the first, and the model confidently approves a return it should refuse.
- Too big and each vector becomes a blurry average of many topics, so the right chunk can lose to a more focused but less relevant one, and the model has to dig through pages of noise.
Some techniques help:
- Overlap. Let neighbouring chunks share a sentence or two, so a rule near a boundary appears whole in at least one of them.
- Split along the structure. Cut at headings, sections and paragraphs, which follow the author’s own idea of what belongs together, rather than every N words regardless of meaning.
- Add context to each chunk, such as the document title and section heading, so a chunk that says “it must be returned within 14 days” still knows what “it” is.
There’s no universal best size: a few hundred tokens with a little overlap is a common starting point, and then you test with real questions.
Key takeaways
- Documents are split into chunks; each chunk gets its own vector and can be retrieved on its own.
- Too small cuts ideas apart; too big blurs the vector and buries the answer.
- Overlap between chunks keeps rules near a boundary whole.
- Splitting along headings and sections, and adding titles to chunks, usually beats blind fixed sizes.
Check your understanding
Go deeper (more technical)
Popular approaches include recursive splitting (try paragraphs, then sentences, then words, until chunks fit), structure-aware splitting for Markdown, HTML or code, and “semantic chunking”, which starts a new chunk where the meaning shifts. Some systems retrieve small chunks for precision, then hand the model the larger section around them for context (“small-to-big” or parent-document retrieval).
Anthropic’s “contextual retrieval” (2024) asks a model to write a sentence of context for each chunk before embedding it, and reported markedly fewer retrieval failures.
The simulation’s retrieval scoring is a simplified illustration.