Prefix caching turns a prompt into shared state

Open an agent that has been live for three months and look at the first eight thousand tokens of every request: system prompt, tool definitions, retrieved documents, conversation history, concatenated in that order. Seven thousand nine hundred of those tokens are identical every time. One sentence in the second paragraph changes, and it carries the current time.

That sentence voids the reuse of everything behind it. A prompt cache matches the prefix byte for byte. The model keeps the processed opening, and the next request that starts identically reads it back at a low rate. One byte differs and everything from that byte onward is recomputed. Nobody receives an alert for this. The bill just gets more expensive, slowly.

What a cache is, in accounting terms

Major providers price it differently and the shape is the same. Writing an entry costs a premium, roughly one and a quarter to twice the standard input rate. Reading a hit costs five to ten percent of the standard rate. Entries belong to one model, so a different model starts a different cache, and entries expire after a period with no reads.

Those three parameters make caching something other than a discount switch. It is an engineering contract with hard constraints on layout: stable prefix, same model, reads closer together than the expiry. Break any one and the savings return as write premium, sometimes more than the savings.

The GenAI working group at the FinOps Foundation states the boundary plainly: if an entry expires before it is reused, the write premium costs more than the caching saves. Which means caching is an investment to be accounted for, not an optimisation to switch on. And what gets accounted for is not the unit price, it is the reuse count.

How the first version usually goes wrong

Most teams write the first prompt on a reading instinct: important information goes first. So the user profile goes first, retrieved documents go first, the timestamp goes first, the request id goes first. The instinct is right for a reader and a disaster for a cache, because it places every per-request value in the most expensive position available.

The worse property is the silence. Cache hit rate drops, the model still answers, latency rises a little, the bill rises a little, and no log line says what happened. By the time the cost curve turns up at month end, the cause is unrecoverable, because five teams edited the prompt over the past three weeks and none of the edits carried a signature.

A rule good enough to work with: put everything that changes at the end. The current user input, the retrieved chunks, the timestamp, the request id. Keep the fixed system prompt and tool definitions at the front, and never insert a per-request value in the middle. The rule is crude and it converts hit rate from an accident into a structure, because as long as the layout holds, the hit rate is predictable.

Tool definitions deserve their own note. They are usually long, almost never change, and are the ideal cached payload. Many teams place them after the prompt, split off by a block of dynamic content, and pay full price for the most reusable bytes in the request. Tool definitions belong next to the system prompt with nothing dynamic between them.

Work out the difference

Take an eight thousand token prefix, fifty thousand calls a day, input at two dollars per million tokens, and cache reads at a tenth of that, twenty cents per million. The numbers come from published pricing pages and vary by vendor. The shape does not.

At a ninety percent hit rate: forty five thousand reads at three hundred sixty million tokens, seventy two dollars; five thousand misses at full price, forty million tokens, eighty dollars. One hundred fifty two dollars a day.

At a forty percent hit rate: twenty thousand reads, one hundred sixty million tokens, thirty two dollars; thirty thousand misses, two hundred forty million tokens, four hundred eighty dollars. Five hundred twelve dollars a day. The difference is three hundred sixty dollars a day, ten thousand eight hundred a month.

At that volume nobody calls a meeting. Put the same difference on eight hundred thousand calls a day and it becomes a hundred seventy thousand dollars a month, the payroll of a mid-sized team. The point is where it comes from. Not model choice, not output length. It comes from where a few lines of changeable text sit in the prefix.

Write premium is still missing from that figure. Every miss triggers a write at around one and a quarter times input price, so the miss side of the bill grows another quarter. A lower hit rate means more writes, and the multiplier chases the team. Low hit rate is not linearly more expensive. It is doubly expensive.

The prefix becomes shared mutable state

Once hit rate reaches a cost report, the prefix stops being one engineer draft and becomes state that several people write at once. A product manager adds a persona line, compliance adds a disclaimer, the retrieval team swaps a document template, security adds a behaviour constraint. Every edit is well intentioned and every edit moves someone else cache bill.

Databases have discussed this for decades under the name shared mutable state, and the answer has never been a reminder to be careful. The answer is three things: versioning, access control, and observation.

Versioning means the prompt exists as a release artifact with an identifier, a change record, and a rollback. Access control means edits pass through one gate instead of six people editing strings in a repository. Observation means hit rate is reported per version and per model, so a drop points at a specific release. Remove one of the three and the other two fail: without version numbers, observation cannot attribute; without access control, version numbers get bypassed.

Teams without this mechanism see a familiar pattern. Hit rate halves in a week and nobody knows why, because five teams edited something last week. That is not a competence problem, it is a missing mechanism for attaching a name to a change. On shared state, attribution matters more than policy, because policy depends on discipline and attribution depends on the system.

Why unit price comparison misleads

The table most buyers build lists every model price per million tokens and picks the cheapest row. That table omits two things: the cache boundary and routing.

When a router sends requests to different models by task, a model switch starts a new cache namespace. The prefix must be recomputed on the new model, and part of the saved unit price is consumed by the recomputation, sometimes all of it. Two optimisations cancel each other while both reports look clean, because one side reports unit savings and the other never books the recomputation as a cost.

The FinOps conclusion is worth quoting directly: the cheapest model per token is not always the cheapest per completed task. The unit only describes a unit. Task success rate and cache hit rate decide how many units get used. Multiply the three and you have the bill. Look at the first two separately and either can point the wrong way.

Our own books

We measured five and a half months of our own calls: one hundred eighty nine thousand four hundred seventeen requests, thirteen point three seven billion input tokens, and five percent of that total written by the model itself. The ratio is not an outlier from one bad week. It comes from a usage pattern that feeds large context in and asks the model to judge.

Input at ninety five percent means the bill is decided on the input side. That side has two levers, length and reuse. Caching only addresses reuse. Filling the window with retrieved chunks loses on length before it can win on reuse, so the two levers move together, length first. Reversed, you watch hit rate climb while the bill stays flat.

We have not pushed caching as far as it goes. We pick models per task, and every switch resets the cache. That cost has no line of its own in our billing, so it can only be inferred from unit price differences. That is the specific gap: we know what we lose in unit price and we do not know what we lose in hit rate, and the second number is usually larger.

Be clear about the nature of these figures. They are a mechanism argument plus our own token composition, not a controlled experiment. We have not run the same load at ninety percent and at forty percent while holding model version, prompt version and traffic mix constant, and that experiment is expensive. So the amounts above are modelled, not measured. Read them as an order of magnitude.

Three actions you can take today

First, split one request prefix and mark every byte that differs on each call. Move all of it to the end. One day of work, visible on the next bill, and no approval needed.

Second, give the prompt a version number and report hit rate by version. The reporting granularity needs to answer one question: which release moved last week number. A report that cannot answer it is decoration.

Third, book model switches as a cost line. Routing is not free. It carries a defined cache penalty that currently sits in a column nobody reads.

Our judgment

Prompt engineering is shifting from writing well to releasing correctly. That is not a rhetorical flourish. A team prompt with no version, no owner and no hit-rate observation is still a personal craft, and at scale it fails, because shared mutable state never degrades gradually. It collapses on one edit.

The judgment makes a testable prediction. If a company AI spend swings more than thirty percent month to month while request volume holds steady, look at the prefix before looking at the model. A model switch leaves a call record. A prefix edit usually leaves nothing at all, which is what version numbers are for.

The reverse also holds: caching early is not automatically right. While the prefix is still changing weekly, caching is mostly write premium and saves nothing. The right moment is after the structure stabilises and changes start going through a process. Run it in the other order and the team concludes that caching does not work, turns it off, and never revisits it.

There is an organisational layer under this. Treating the prefix as a release artifact means someone owns it, usually the platform team rather than the product team. That is not centralising power for its own sake. It gives shared state a visible author. State with no author gets edited by everyone and owned by nobody.

A question for whoever holds the cost number: of last quarter AI spend, how much came from model choice and how much from layout. The first has a dashboard. The second has a bill, and the bill does not explain itself.

References

External material comes from the pages below. Internal figures come from our own call statistics.

FinOps Foundation, GenAI FinOps: How Token Pricing Really Works (updated October 28, 2026: cache write premium, entry expiry, caching that costs more than it saves, and why the cheapest model per token is not the cheapest per task)

Economize, AI Token Economics 2026 (vendor blog: Sonnet 5.5 at two dollars per million input, ten per million output, cache reads at five percent; this article uses a conservative ten percent read rate)