Prompt caching saves on price, not on history

Every six hours those scheduled jobs wake up and do the same thing first: they resend the whole conversation from the opening line. System prompt, tool definitions, last round’s conclusion, the failed attempts in between, all of it goes back into the model unchanged. We opened 30 days of call records, and what we saw was not one bill. It was two streams.

One: the discount lands on the rate

Input has two price tiers. One covers what is genuinely new in this request, billed at the cache-miss rate. The other covers a prefix the model has already processed and can reuse, billed at the cache-hit rate. The spread between those two rates runs from 10x to 50x, and the volume spread between the streams can reach an order of magnitude.

DeepSeek’s published table prices cached input at $0.003 per million tokens against $0.15 for a miss, a 50x difference. OpenAI’s documentation charges cache reads at 0.1x the input rate and cache writes at 1.25x. Anthropic documents reads at 0.1x, with some models at 0.025x and 0.05x, and writes at the same 1.25x.

That price structure explains something teams routinely miss: the discount lands on the rate, not on the volume. Not one character of the history you resend disappears. Most of it simply costs less. What actually drives the bill is how much history each round has to carry and how many rounds a single task takes.

Two: what our own 30 days measured

Between 30 August and 29 September 2026 this system made 3,159 model calls: 389 million fresh input tokens, 3.70 billion cached input tokens, and 60.4 million output tokens.

Converting that mix at DeepSeek’s off-peak rates gives $58.4 for fresh input, $11.1 for cache hits, and $36.3 for output, roughly $105.7 in total. Cached volume ran at 9.5x the fresh input while costing 19% of it.

Two gaps are worth stating plainly, because they both push in the same direction. Our own records leave the cache-write counter empty, so the 1.25x write premium never enters these numbers; and part of the traffic runs on subscription channels, which count toward volume but not toward the per-token conversion. Real cost sits above this estimate, never below it.

Three: the ratio follows from session length

The cached stream can outgrow the fresh stream by an order of magnitude because it accumulates. Assume a task resends its full history each round and each round adds a comparable amount of new content. A session of n rounds then sends n segments of fresh input, while cumulative cache reads run from zero up to n minus one, which is about half of n squared. Divide one by the other and the volume ratio comes out near (n minus 1) divided by 2.

Feed our own reading into that: 9.5x corresponds to an average session of about 20 rounds. Run it the other way and a target ratio of 4x requires an average session of roughly 9 rounds. Split the same task into two sessions of 10 rounds and the cumulative reads come to about 90 segments on both sides, against 190 for a single 20-round session, with fresh input unchanged and cached volume nearly halved.

The assumption has to be stated because equal-length rounds are not true in real data; early rounds in a long session tend to be larger. The numbers above are order-of-magnitude estimates for choosing a direction, not for pricing. Their value is that they convert a vague sense that something is expensive into a variable you can act on, which is session length.

Four: cache lifetime is measured in minutes

A cache has a retention window, and a shorter one than most teams assume. Vendor documentation puts the default at five minutes, with the write side paying 1.25x the input rate for reuse inside that window. Longer retention exists at some providers at a different price, and is worth checking provider by provider.

Our scheduled jobs wake up every six hours, so the gap between two executions is far past five minutes. A hit across runs is effectively impossible, and hits can only happen among the rounds inside one session. That also explains the ratio: every round inside a session resends the full history from the opening line, and the more rounds a session has, the longer the cumulative reads become.

The architectural consequence is direct. The boundary of the saving is not between tasks, it is inside a session. Splitting a task into shorter sessions is more realistic than expecting two tasks six hours apart to share a cache.

Five: hit rate is not a cost metric

A hit rate is a ratio. Cost is an absolute quantity. The denominators differ, and managing an absolute quantity with a ratio produces a familiar illusion: 99% sounds finished.

Run the arithmetic on what falls through. If 10% of those 3.70 billion tokens missed, that is 370 million tokens billed at the miss rate: $55.5, nearly the same as the cost of every fresh input token we sent in the whole month, $58.4. A hit rate sliding from 99% to 98% looks like one point; the volume that misses grows by 37 million tokens.

A shorter version is easier to keep: every 10% that leaks out of the cached stream makes you pay for the fresh input stream twice. The larger the stream, the more expensive that conversion gets, which is why long-conversation teams should watch leakage rather than the ratio.

Six: why a cache misses

Matching is byte for byte. Anthropic’s documentation is blunt about it: a hit requires a prefix that is 100% identical, including every piece of text and every image up to the block carrying the marker. Writes only happen at breakpoints, and content past the last breakpoint is not cached at all.

There is also a window. The same documentation walks through a case where a growing conversation pushes the breakpoint more than 20 content blocks past the last write; the lookback finds nothing there, and the whole prefix is reprocessed. Cache reach is not unlimited. It follows the breakpoint.

The most common self-inflicted miss is a different one: something that changes every round, buried inside the prefix. The documented example is a timestamp. It changes the prefix hash on every request, so the lookback never matches, and the cache behaves as if it were never enabled. A freshly generated identifier or the incoming user message has exactly the same effect.

Our rule follows from that: anything volatile belongs after the breakpoint, and anything stable (the system prompt, tool definitions, reference documents that rarely change) belongs before it. The rule costs nothing to follow, yet it is easy to get wrong on the first attempt, because the most natural place to put a timestamp is the first line of the system prompt.

Seven: output is the counter-intuitive line item

Output came to 60.4 million tokens, one sixty-first of the cached stream by volume, and it still accounted for a third of the bill. Output is priced at 200x the cache-hit rate: $0.60 against $0.003 per million tokens.

That has a direct engineering consequence. Whether a model can stay quiet matters more to your bill than how cheap its tokens are. Token rates are something you negotiate; output length is something you control. Writing a ceiling for each task pays back faster than shopping between providers.

The mistake we made was leaving output length to the model’s judgment. A task description asking for something brief reliably returns a reply with section headings, a summary, and recommended next steps, three times longer than the substance requires. A number, such as “no more than 300 words, and say why if you exceed it,” changes the result immediately.

Eight: compression is not free either

Compacting history costs a call of its own: it reads the span being compacted and writes a summary back, and output is the most expensive tier on the price list. A compaction that reads 100,000 tokens and writes 2,000 comes to $0.015 plus $0.0012, about $0.016.

Whether that money comes back depends on how many rounds remain. A shorter prefix saves tokens only in the rounds that follow, so a task with one round left is better off uncompacted. We therefore write two thresholds: how large history may grow before compaction triggers, and how many rounds must remain before compaction is allowed to run. The second rule is the one teams skip, and it blocks the worst case, compacting history to serve a single final step.

Two configurations make the arithmetic visible. The same task run for 20 rounds without compaction sends 20 segments of fresh input against roughly 190 segments of cumulative cache reads. Split the same task into four sessions of five rounds and the cumulative reads come to about 40 segments across all four, while fresh input stays at 20, at the price of context dropped between sessions. A third option keeps the long session and compacts every five rounds, paying for four compaction calls, about $0.065.

What separates those three is not model choice. The first pays to resend history, the second pays for context lost between sessions, and the third pays for the compaction calls themselves. Which one fits depends on the shape of the task: work whose steps depend on every previous step belongs in the third, and work whose steps can be accepted independently belongs in the second. That is an architecture decision rather than a prompt technique, and it moves the bill further than switching to a cheaper model does.

Nine: what a 90% price cut does not fix

Gartner forecast in March 2026 that by 2030 inference on a comparable model will cost over 90% less than in 2025, and that 2030 models will be up to 100 times more cost-efficient than models of similar size built in 2022. The forecast will most likely hold.

It lands on the rates. The volume ratio between the cached stream and the fresh stream is set by how much history each round carries, and price has no influence on it. If history grows along with the number of tasks, part of the cut gets absorbed by volume. Our judgment: a price decline changes the scale of the bill, not its shape, and whoever controls history volume first collects the entire benefit of the decline.

Ten: three knobs, in one order

The first knob is a compression threshold: how large history may grow before it must be compacted, and how much survives compaction. It sets the volume ratio between the two streams.

The second is breakpoint placement and stability: stable content first, volatile content after. The number of breakpoints is capped, four writes per request in OpenAI’s documentation, so they cannot be scattered around. Two or three fixed positions are enough.

The third is an output contract: a ceiling per task, with a stated reason required to exceed it. This one governs the most expensive tokens on the bill.

The order matters: measure first, then fix volume, and only then negotiate rates. Volume is a divisor and rate is a multiplier, so do the division first. Opening with a rate negotiation hands a variable you can control to a vendor.

Eleven: when caching does not pay

Caching is not automatically cheaper. Writes cost 1.25x the input rate, and a prefix has to be read a second time to earn that back, since the read costs 0.1x and the two requests together come to 1.35x against 2x without caching. The break-even is low, but it means a single round trip should not enable caching at all; the 25% premium never returns.

One security boundary is worth noting as well. OpenAI’s documentation suggests separate cache keys per user to prevent cache-hit probing, where someone uses trial requests to infer whether another user’s context is cached. In multi-tenant deployments the cache key should follow the tenant, never a global default.

Twelve: four fields and three questions

To read your own bill properly you need four fields per call: fresh input tokens, cache-hit tokens, cache-write tokens, and output tokens. Multiply by the number of calls a task takes and you have unit cost per task. Those four fields are far more useful than a monthly total, because they are the only version of the number that points at an action.

Three questions are worth asking a platform before you sign or renew. How many times the input rate are cache reads and writes, and how many write breakpoints does one request allow? At what granularity is a hit decided, how wide is the lookback window, and can a hit cross sessions? Can the bill be split by task, rather than a single total by key or by day?

Next month we are changing exactly one thing: compression moves from judgement calls to a threshold, breakpoints get pinned to stable prefixes, and we watch whether leakage out of the cached stream falls from 10% below 5%. How many times your fresh input is your cached stream?

References

• DeepSeek pricing table (cache hit $0.003, miss $0.15, output $0.60 per million tokens, off-peak)
• OpenAI prompt caching guide (reads at 0.1x, writes at 1.25x, up to four write breakpoints, separate keys against hit probing)
• Anthropic prompt caching docs (byte-exact matching, five-minute retention, 20-block lookback window, the timestamp that breaks the prefix hash)
• Gartner forecast, 25 March 2026 (inference over 90% cheaper by 2030 than 2025)
• Epoch AI on long-run inference price trends measured against benchmark scores

The call counts, token volumes, and ratios above come from our own model-usage records for 30 August to 29 September 2026 (3,159 calls) and are not extrapolated. Dollar figures use the public off-peak list rates cited here and are estimates, not settled amounts.