Context rot and "lost in the middle": longer context isn't always better
TL;DR
Lost in the Middle (2023) found models retrieve best at the start and end of context; Chroma's Context Rot study of 18 models found quality drops with input length even on simple tasks. Feed less but relevant context, remove distractors and test at real lengths.
LLM context windows now reach hundreds of thousands, even millions, of tokens. The temptation is huge: stuff in every document and let the model find what it needs. But two studies worth reading show the opposite: models don't use every part of the context equally, and quality drops as input grows, even on very simple tasks. This phenomenon is called context rot.
Study 1: Lost in the Middle (2023)
Liu et al. (2023) tested two tasks: answering questions over multiple documents, and retrieving a value by key from a long key–value list. They varied the position of the needed information in the context.
- The results formed a U-shape: models did best when the information was at the start or the end of the context.
- When the information was in the middle, performance dropped noticeably.
- The effect appeared even in models marketed as "long-context".
Study 2: Context Rot (Chroma, 2025)
Chroma's research team tested 18 models from Anthropic, OpenAI, Google and Alibaba on controlled tasks:
| Finding | What it means |
|---|---|
| Longer input reduces reliability, even for simple copying or retrieval | You can't assume "in the context = used" |
| The less the question overlaps in words/meaning with the target passage, the faster performance falls with length | "Needle in a haystack" benchmarks (keyword matching) are too easy compared with reality |
| Distractors (similar but wrong passages) reduce accuracy; more distractors reduce it further | Old versions and drafts in the context are dangerous |
| Shuffled filler text gave better results than coherent filler text | Context structure affects results in hard-to-predict ways; measure it |
| LongMemEval: a focused prompt (~300 tokens) beat the full prompt (~113k tokens) for every model family | Selecting the right context usually beats including everything |
The report also notes that model families react differently to distractors: Claude models hallucinated the least and often abstained when unsure, while GPT models hallucinated the most in the experiment.
Tactics against context rot
- Select before loading: use retrieval (RAG) or a filtering step so only relevant parts go in.
- Place things well: long documents at the top, the question and key instructions at the end.
- Remove distractors: delete old versions, duplicates, irrelevant logs; clearly mark which source is authoritative.
- Add structure: wrap each document in named tags with metadata so the model can locate it.
- Quote first, answer second: ask the model to extract relevant passages before reasoning.
- Clean up context in long sessions: for agents, clear old tool results or summarise periodically instead of letting it grow forever.
Risks and caveats
- Results depend on model and task: these are trends, not fixed numbers. Newer models may improve; measure for yourself.
- Over-trimming also hurts: filtering too aggressively loses needed information. Balance it with evals.
- Double cost: long context is both more expensive and potentially less accurate, so "stuff it in to be safe" is a bad choice on both counts.
How to apply it
- Dev: Test at your product's real input lengths, placing the target information at different positions, and see whether accuracy drops in the middle.
- Tech Lead: Put a "context budget" into the design: how many tokens each kind of data may take, and what gets priority when over budget.
- PM: Don't market a feature as "reads 1 million tokens". Measure and promise on answer quality at realistic lengths.
Sources & references
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al. (arXiv)
- Context Rot: How Increasing Input Tokens Impacts LLM Performance — Chroma Research