The NewsRupt

Context length is the easiest specification for a model vendor to advertise and one of the hardest for a buyer to evaluate. A million tokens is a number anybody can compare. Whether the model can reliably find the one paragraph that matters inside that million is a different question, and it is not answered by the specification.

The distinction is between capacity and retrieval. Capacity is how much text the model will accept without error. Retrieval is how dependably it uses any given piece of that text when the answer depends on it. The two diverge, and they diverge in ways that are uneven rather than gradual.

Public and private testing has converged on a broadly consistent picture. Accuracy on a fact placed at the very start or very end of a long input remains high. Accuracy on the same fact placed in the middle degrades, sometimes sharply, as the input grows. Tasks that require combining several facts scattered across the window degrade faster than tasks requiring a single lookup. And a model asked to reason over a long document will often produce a fluent answer built on the portions it attended to, with no signal that it missed the rest. That last property is the operationally dangerous one: the failure is silent.

For anyone building on top of these models, the consequence is that the long window is best understood as headroom rather than as a strategy. It removes a class of engineering pain — the brittle chunking logic, the arbitrary truncation, the constant tuning of how much to include — and that is genuinely valuable. It does not remove the need to decide what belongs in the prompt.

The teams getting the most from long windows tend to use them in a particular way. They still retrieve, but less aggressively: rather than fetching five tight snippets, they fetch twenty generous ones and let the model sort it out. They place the material the answer most likely depends on near the end of the input, where attention is empirically strongest. They ask for citations to the supplied text and check that the cited span actually exists, which catches a large share of failures cheaply. And they test retrieval on their own documents at their own lengths, because performance varies by document structure in ways no published benchmark will capture.

There is also a cost dimension that the specification obscures. Filling a large window is not free, and it is not free in latency either. A request carrying four hundred thousand tokens of context costs what it costs on every call, whether or not the extra material was used. Prompt caching changes this arithmetic substantially where the bulk of the context is stable across calls, and barely at all where it is not. A system that stuffs the window by default because the window exists tends to be both slower and more expensive than one that selects, without being measurably more accurate.

The honest limitations here matter. Public long-context benchmarks are mostly synthetic — a fact inserted into filler text — and real documents are not filler; they contain related material, contradictions and near-duplicates that make retrieval harder in some ways and easier in others. Results vary considerably between model families and between versions of the same family, so a finding about one model in one month should not be generalised. And vendors are actively working on exactly this weakness, so the gap between advertised and usable context is narrowing, unevenly.

The claim worth holding is modest. Advertised context length tells you what the model will accept. It does not tell you what the model will use. Until a buyer has tested retrieval on their own corpus, at their own lengths, with their own questions, the headline number is a compatibility statement rather than a capability one — and treating it as a capability is how systems end up confidently summarising documents they only partly read.