Long context vs RAG
cache hit rate decides

A bigger window is not the same thing as a smaller bill. What decides whether long context beats retrieval is how much of the corpus one request actually needs, and how often the same prefix comes back inside one cache window.

The cache price line says two things, and only one gets quoted

Vendor pricing pages reduce caching to a clean discount: reads bill at ten percent of the input rate, writes at one and a quarter times for a five minute window or twice for a one hour window. The first half of that sentence travels well, which is why it is everywhere.

The second half is where the resident-context invoice comes from. Putting a corpus in the window is not a one time purchase. It is a write that repeats every cache window. A five minute window rewrites the prefix twelve times an hour; a one hour window writes it once, at double the price. Those two numbers together set the floor cost of a resident design, and that floor does not care how much traffic you have.

There is a third line worth reading closely. Long context is a premium tier in several vendor price books, not the standard one: input is charged at a higher rate past a length threshold, and the model compacts long histories on its own. Vendors setting those two behaviours as defaults is itself a judgement about reliable context. If a window could be filled and used, neither the surcharge nor the compaction would need to exist.

Read together, the three lines invert the intuition. Caching cuts the cost of reading, while the fate of a resident design is decided by the cost of writing and by how many times the prefix is read. The steeper the read discount, the stronger the vendor signal that you are expected to reuse one prefix many times.

The crossover comes down to three numbers

Write both architectures as cost functions and the crossover can be solved directly. Let the corpus be N tokens, the input rate P per million tokens, the material one request actually needs K tokens, and the number of requests arriving inside one cache window r.

• Resident, per window: a write of N times the write multiplier, plus 0.1 N for every read.
• Retrieval, per window: r times K, because only the retrieved slice enters the window.

Setting the two equal gives the critical reuse count: the write multiplier times N, divided by K minus 0.1 N. That expression hands back a veto first and a quantity second.

The veto sits in the denominator. When K is smaller than 0.1 N, meaning one request needs less than a tenth of the corpus, the denominator turns negative and resident context cannot win at any reuse rate. The reason is plain once stated: retrieval pays only for the slice it actually used, while a cache read is billed at ten percent of the entire corpus. The bigger the corpus and the narrower the query, the wider that gap gets.

Take a concrete case. A forty million token knowledge base, a mid tier model at two dollars fifty per million input tokens, and eight thousand tokens retrieved per request. Retrieval costs two cents per request. A resident read costs 0.1 times forty million tokens, which is ten dollars. That is a factor of five hundred, and it does not close as volume grows. It widens.

The quantity lives in the numerator. Once K clears 0.1 N, the critical reuse count equals the write multiplier divided by the share K takes of N, minus 0.1. Three corpus shapes land in very different places.

• A four hundred thousand token contract set where reasoning needs three hundred thousand of it, about seventy five percent: the one hour crossover is roughly three reuses per hour. Contract review asks a dozen questions of the same document inside an hour, so resident wins comfortably.
• A one million token product library where a request needs two hundred thousand tokens, about twenty percent: the crossover is exactly twenty reuses per hour. That is a high bar for most internal tools. A nightly batch clears it, scattered daytime questions do not.
• A forty million token ticket and manual corpus where a request needs eight thousand tokens, two hundredths of a percent: resident context is arithmetically dead unless the retrieval layer disappears entirely and every request carries hundreds of thousands of tokens.

Comparing the two cache durations adds a second counterintuitive result. The five minute window has the cheaper write multiplier, one and a quarter against two, but it is one twelfth as long, which makes holding one prefix resident about one hundred and fifty times more expensive per hour. The cheap cache tier is built for small prefixes touched constantly, not for keeping a corpus warm.

Dividing reuse by window length exposes a third common misread. Twenty reuses per hour sounds modest, but it is an average rather than a peak. Traffic that lands in one nightly batch clears the gate easily inside a single one hour window. Traffic spread evenly across the day, say one call every three minutes, lands almost exactly on twenty per hour and only just qualifies, while the same spread under a five minute window works out to twelve reads an hour against a requirement of one hundred and fifty. One design holds for batch work and fails for steady arrival, and the difference is neither corpus nor model. It is arrival shape, which is why the measurement section puts the request time histogram first.

Write cost carries one more line that is easy to omit: corpus updates. A cache requires a byte identical prefix, so any edit invalidates the whole prefix and forces a rewrite. A knowledge base that takes a thousand record updates a day, spread through working hours, can end up rewriting inside most windows, and the write line then scales with windows times corpus size while staying completely detached from request volume. Retrieval handles the same situation by rebuilding the small slice that changed, which is a large part of why frequently updated corpora are rarely good resident candidates.

Winning the cost argument is not the same as answering correctly

The crossover answers half the question. The other half is whether the model can still find the passage you wanted once the whole corpus is in the window.

Chroma’s context rot work found degradation starting before the window fills rather than at the moment it tops out. Public sequential retrieval benchmarks put a number on it: across contexts from eight thousand to one hundred and twenty eight thousand tokens, the best of six models reached sixty three and a half percent, so even the strongest model missed more than a third of ordering dependent retrieval tasks. Vendor side multi needle tests point the same way, with single needle accuracy near ninety five percent collapsing to around sixty percent when several needles have to be recovered.

This is not a defect that the next model generation retires. It follows from how attention behaves over long sequences, so the curve improves gradually rather than jumping. Third party analysis puts the practical band at one hundred and twenty eight thousand to two hundred thousand tokens, past which multi needle work stops being dependable for most models. Vendor defaults agree: a model with a million token window triggers automatic compaction at one hundred and fifty thousand, and two other providers move to higher input pricing past two hundred thousand and two hundred seventy two thousand tokens.

The working ceiling for a resident design is therefore not the number on the product page. It is the safe budget your own corpus and your own questions measure. That number can only be measured locally, because rot speed depends on how hard the retrieval task is. Ask multi hop or multi needle questions of the same corpus and the budget shrinks.

Deleting the retrieval layer takes two other things with it

On a cost sheet the retrieval layer is pure expense. In production it is two pieces of infrastructure: citation provenance and a permission boundary.

Provenance is not optional in regulated work. With retrieval in place an answer points back to the passages that were scored, so an auditor can reconstruct the input. Move to a resident corpus and the input is the whole document set, which drops citation precision to knowing what the model saw rather than which passage the answer rests on. That gap is hard to close after an incident, because the attention pattern of a past run cannot be rebuilt.

The permission boundary is the quieter loss. A retrieval layer is a filter by construction: who may see which passage is resolved before anything enters the window. A resident design puts the corpus into the prefix wholesale, so visibility has to be enforced upstream instead, one cache prefix per visibility group. That multiplier lands directly on the write cost, and it is almost never in the comparison deck.

Three architectures, three measurable gates

The choice then reduces to three questions with numeric answers, rather than a matter of taste.

Our position

The long context against retrieval argument is being had at the wrong layer. The debate is usually framed as which technique is more advanced, while the cost function turns on the share of the corpus one request needs and the number of times one prefix is reused inside a cache window. Window size is neither the numerator nor the denominator.

Second, the cache discount is widely misread as making resident context nearly free. Reads are cheap, but the resident invoice is mostly writes, and writes repeat every window. Add normal corpus updates and the write side grows again, because any edit rewrites the whole prefix.

From those two readings, our default instruction is to measure before migrating. Most enterprise corpora are queried narrowly, K over N is small, and in that class resident context has no arithmetic path to winning. Where measurement shows heavy reuse and requests that need most of the corpus, the case for migrating has nothing to do with window size. It is that retrieval was doing unproductive work for that workload all along.

One last judgement belongs to the compliance lead rather than the platform owner. In most mature architectures the retrieval layer is also the evidence layer. Removing it saves retrieval spend and reintroduces the cost as audit traceability, which vendor material tends not to mention and procurement review tends to ask about first.

Three measurements, two weeks, no migration

Every claim above can be checked against data you already have.

1. Pull thirty days of request timestamps and compute the median and ninety fifth percentile of arrivals per hour. That distribution caps reuse, and it decides whether the five minute or the one hour cache is even worth discussing.
2. Run one multi needle retrieval test on your own corpus, no public leaderboard, and record the length at which accuracy starts to fall. That is your safe budget, and it is typically well under the advertised window.
3. Add a cache instrument to the existing path that logs written and read tokens separately. With a real hit rate in hand, the effective input price becomes comparable with the per request cost on the retrieval side.

With those three numbers the crossover formula has everything it needs. The result either shows a migration with a defensible business case, or shows the retrieval layer costing far less than resident context. Both outcomes beat arguing about window sizes.

References

• Anthropic cache write and read multipliers, one and a quarter times or twice for writes, ten percent for reads: anthropic.com/pricing
• Third party summary of cache multipliers and batch discounts across providers: flexera.com/blog/ai/prompt-caching-breakdown
• Context rot report, degradation beginning before the window fills: research.trychroma.com/context-rot
• Sequential retrieval benchmark, best of six models at sixty three and a half percent between eight thousand and one hundred and twenty eight thousand tokens: arxiv.org/abs/2504.04713
• Vendor compaction defaults, long context surcharge tiers, multi needle decline and the one hundred and twenty eight thousand to two hundred thousand token practical band: glasp.co/articles/context-rot-rag-long-context-hybrid
• Vendor side test of multi needle accuracy falling from ninety five to sixty percent: openlayer.com/blog/needle-in-haystack-ai-testing-llm-context-retrieval
• Industry analysis of long context and retrieval used together: atlan.com/know/llm-context-window-limitations