Contextual Retrieval: fixing RAG's root problem

RAG (Retrieval-Augmented Generation) is the most common way to make an LLM answer from your documents: split documents into chunks, find the relevant chunks, then put them in the prompt. The term was coined in a paper by Lewis et al. (Facebook AI, 2020). The classic problem: once split, each chunk loses its context. A passage saying "Revenue grew by 3% over the previous quarter" doesn't say which company or which quarter, so search misses it. Anthropic proposed Contextual Retrieval to close exactly this gap.

The idea: attach context to each chunk

Before embedding and indexing, have Claude read the whole document together with each chunk, then generate a short explanation (usually 50–100 tokens) prepended to the chunk:

Before: "Revenue grew by 3% over the previous quarter."
After:  "This chunk is from ACME's Q2 2023 report, discussing
         quarterly revenue; Q1 revenue was $314 million.
         Revenue grew by 3% over the previous quarter."

The contextualised chunk feeds both indexes: Contextual Embeddings (search by meaning) and Contextual BM25 (search by keyword).

Anthropic's reported results

Measured by retrieval failure rate (1 − recall@20), averaged across domains:

The reranking setup in the experiment: retrieve the top 150 chunks, rerank, keep the best 20. Passing 20 chunks to the prompt worked better than 5 or 10.

Is the cost worth it?

Generating context for every chunk sounds expensive, but thanks to prompt caching (the document is loaded once and chunks reuse the cache), Anthropic estimates a one-time cost of about $1.02 per million document tokens. That's an indexing cost, not a per-query cost.

The full pipeline

1. Split documents into chunks (a few hundred tokens; consider boundaries and overlap).
2. For each chunk, call Claude (a small model, with the document cached) to generate context.
3. Embed "context + chunk" and build a BM25 index over the same text.
4. At query time: search both indexes in parallel, merge results, rerank, take the top K.
5. Put the chunks in the prompt and require source citations.

Risks and caveats

• Sometimes you don't need RAG: per Anthropic, a knowledge base under about 200,000 tokens can go straight into the prompt (with caching), which is much simpler.
• Customise the context prompt: adding a glossary for your domain usually beats a generic prompt.
• Reranking adds latency and cost to every query; measure the trade-off on real traffic.
• More chunks can distract: the optimal K must be found with evals, not chosen by feel.

How to apply it

• Dev: Build an eval set of 50–100 questions with their correct chunks; measure recall@20 before and after adding context, and only then add BM25 and reranking.
• Tech Lead: Run context generation via the Batch API with caching to cut indexing cost; store the context separately so it can be reused.
• PM: When an internal chatbot "can't find" information that really is in the docs, it's usually a retrieval bug, not a model bug. Fix retrieval first.