Chunking

Chunking is the step where a body of text is cut into smaller pieces so each piece can be indexed and retrieved on its own. It is the least glamorous part of a retrieval system and routinely the part that decides whether it works.

Anthropic's description of a standard pipeline puts it first: break the corpus into smaller chunks of text, usually no more than a few hundred tokens; use an embedding model to convert those chunks into vectors that encode meaning; store the vectors in a database that supports searching by semantic similarity. At query time, the nearest chunks are found and added to the prompt sent to the generative model.

The context problem

Cutting text up destroys the thing that made it understandable. Anthropic states the failure directly: traditional retrieval removes context when encoding information, which often results in the system failing to retrieve the relevant information at all.

Its example is the clearest one available. Imagine a collection of financial filings and the question "What was the revenue growth for ACME Corp in Q2 2023?" A genuinely relevant chunk might read: "The company's revenue grew by 3% over the previous quarter." That sentence names neither the company nor the quarter. Nothing about its vector will bring it back for a query that names both, and if it does come back, the model has no way to know what it refers to.

This is not an exotic edge case. It is what happens to most pronouns, most "this", most section-relative references and most tables whose header row landed in a different chunk.

Contextual retrieval, and what it measured

The fix Anthropic published is preprocessing rather than clever querying: prepend chunk-specific explanatory context to each chunk before embedding it and before building the lexical index. The transformed version of the example reads: "This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter."

Because annotating millions of chunks by hand is impossible, the context is generated by a model, prompted to situate each chunk within its whole document and answer with nothing else. The resulting text is usually 50 to 100 tokens.

The reported results, measured as one minus recall at 20 across codebases, fiction, ArXiv papers and science papers:

Three of Anthropic's implementation notes are worth carrying over regardless of stack: chunk size, chunk boundary and chunk overlap all affect performance; some embedding models benefit more than others; and passing 20 chunks to the model outperformed 5 or 10 in their tests, though more information can also distract a model, so there is a limit.

The part that is a writing problem

An engineer can tune chunk size. Nobody can tune a document into being self-contained. If a section only makes sense after the two above it, every chunking strategy will produce a piece that means less than the author intended.

That is why chunking belongs in a marketing glossary at all. The way your page is written decides what its chunks say once they are separated from it:

• One subject per section. A section covering four topics produces a vector that means none of them.
• Headings that state the claim, not headings that tease it, because the heading is often the only context a chunk keeps.
• Repeat the subject. Naming the product, the version and the period inside the paragraph rather than relying on the H2 above it is the manual version of contextual retrieval.
• Keep identifiers literal. Exact strings are what a lexical index can match when a vector cannot.
• Do not split a claim from its qualifier. A caveat in a different paragraph is a caveat in a different chunk.

When not to bother

The first question is whether retrieval is needed at all. Anthropic's guidance is that a knowledge base under 200,000 tokens — roughly 500 pages — can simply go in the prompt, no retrieval required, and notes that prompt caching makes that approach cheaper and faster than it used to be. Chunking is the price of scale, not a virtue, and a system that pays it for a corpus that fits in the context window has bought a failure mode for nothing.