Our agent token bill
95% is rereading

At eleven at night, with the site build and the monitors finished, I opened this machine’s state database and ran one aggregate query. I wanted a number I had never actually computed: across half a year of paying for model calls, what did the money buy. The answer came back as 189,417 API calls, 13.4 billion prompt tokens on the input side, and 155 million output tokens. The model’s own words were 5.01 percent of the bill.

The other 95 percent is material resent every single turn: the system prompt, the company rules, the skill documents, the tool list, the conversation history, the task for this run. The model remembers nothing between calls, so each call rereads all of it from the start. That is not a flaw in any one vendor’s implementation. It is the shared premise of every chat-style interface shipping today.

I expected this to be a cost story. Once the numbers were on one screen it turned into an architecture story: what sets an agent’s bill is not the price per token, it is how many bytes of the context are guaranteed to change on every call.

What one call actually sends

Definitions first, because these numbers get misread otherwise. The usage table stores one row per session-and-model combination, not one row per request. Ten thousand nine hundred and sixty-seven rows accumulated to 189,417 underlying calls. New input was 2,944,657,059 tokens, cache reads were 10,425,268,352 and are counted separately, and the two together give the 13.4 billion prompt total. Output was 155,340,340 tokens, of which 68,754,534 were reasoning tokens.

Divide it out: 15,546 input tokens and 820 output tokens per call, nineteen to one. That ratio makes a decent diagnostic. Measure three to one and your prompt is thin, so rereading is not your main cost. Measure forty to one or worse and most of your money is spent having the model read the same material again, in which case switching to a cheaper model gives you a proportional discount and changes nothing structural.

The output side deserves its own line. Reasoning tokens are 44.3 percent of output, 363 per call on average. Nobody sees them and they are billed all the same. The point is not whether reasoning models are good. The point is that it breaks a common optimization: trimming the visible answer. When nearly half of what you pay for on the output side is internal computation rather than the text you asked for, the ceiling on word-count savings is fixed. You can only cut from the other end, by reducing the calls that need to think.

Caching took rereading down to a fifth, not away

Cache reads came to 10.4 billion tokens, 78 percent of total prompt length. Read as a waste figure that looks terrible. Read against how caching is priced, it is the good news: Anthropic’s prompt-caching documentation states that cached portions are billed below the normal input rate. So 78 percent does not mean we wasted 78 percent of the money. It means 78 percent of the rereading came at a discount.

Why the hit rate is that high. The prefix is stable. System prompt, rule files, skill text, tool definitions: same order, same bytes, every call. Only the task at the tail and the newest messages change, and cache matching keys on exactly that byte-identical prefix. Change one space, inject a timestamp, reorder the tool list, and the whole cached segment is void.

That constraint flips a piece of common advice. The lever is not shortening the context, it is shrinking the part that must change. Cut an 8,000-token rule file to 3,000 and keep its content in flux and you save little. Compress a 40,000-token history while the prefix stays byte-stable and the saving is real. The first move makes the fixed part smaller. The second makes the changing part smaller. Most teams do the first, because it is the one you can see in a document.

Half of our own setup confirms it. Rules are split per unit and skills load on demand. Both were introduced so the model would not be distracted by irrelevant text, not to save money. The cost effect turned out larger than expected, and the reason is that both shrink the changing part: a wakeup loads only the few hundred lines it needs this time, everything else stays out of the window.

A 3.7 GB state store, and one detour I cannot put a number on

The state database on this machine is now 3.7 GB: 540,468 messages across 11,166 sessions. Of those messages 19,577 are marked compacted, 3.6 percent. That share is low, and it means most messages sit there in full, resent turn after turn.

Compression has a bad middle. Compress too early and the context loses its antecedents, so the agent goes back to the files and searches again for what it just read: the extra calls cost far more than the tokens saved. Compress too late and every turn carries a long history, compounding as it goes, because the later you leave it the more expensive each turn becomes and the more you want to postpone it. The real test is not a token count. It is how many items in this session remain unfinished and whether the finished ones left their conclusions in a memo. We moved the trigger from “window eighty percent full” to “write the conclusions, then compress,” and the waste on the output side dropped noticeably.

One honest gap. I have no clean measurement of input per turn before and after compression. Sessions do different work, no A/B was run, and averaging them together would produce a number that misleads. So this section gives the mechanism and no percentage. If it reads well but does not survive arithmetic, it does not go in.

Ninety percent cheaper tokens, unchanged bills

In October 2026, Cockroach Labs cited a Gartner projection on its blog: inference on a trillion-parameter model will cost roughly 90 percent less by 2030. The same post argued enterprise bills will not fall with it, for three reasons: an agent burns far more tokens per task than a chat exchange; consumption grows faster than unit price falls; and providers do not fully pass reductions through. That is a vendor blog relaying a third-party forecast, and I label it as relayed rather than established fact.

The second reason explains our numbers exactly. Make the token ninety percent cheaper while the material reread per call goes from fifteen thousand to a hundred and fifty thousand and the bill rises tenfold. Agent history grows by nature: longer tasks need longer context, more tools, more retrieval results. Price and consumption run on two different curves, and the daily dashboard only shows the first.

Microsoft’s Azure blog states the mechanism plainly: everything in the context window is billed every turn, so redundant content is charged repeatedly. Its own contrast is that a chatbot answering one question can ignore this cost, while an agent working across many turns toward one outcome usually finds it the largest expense. Sourcegraph’s engineering blog adds a second angle: with dense attention, cost grows with the square of sequence length, and longer context also lowers recall. Their reported experience is that a 100,000-token codebase summary performs worse than a targeted 5,000-token retrieval. All three are vendor sources, technical explanation and product position rather than independent testing. They describe one mechanism, though, and our 13.5 billion tokens have the shape that mechanism predicts.

Four moves, ordered by whether you can do them today

First, pin the prefix. Lock the order and content of the system prompt, rule files and tool definitions, and move every timestamp, session id and random seed to the tail. This is engineering discipline, needs no vendor cooperation, and costs nothing. To check whether it worked, send two calls of the same kind and watch whether the cache hit rate jumps on the second. If your platform will not show you that number, that is a procurement gap on its own.

Second, make conclusions the compression trigger rather than length. Before compacting, have the model write three lines on the current state and what has been decided, then paste that at the top of the compressed context. Details can go; the decision chain stays, and the agent stops rereading to rebuild it. We replaced the length threshold with this, and the effect is measurable: fewer calls immediately follow a compression than before.

Third, move deterministic work out of model calls. Any job whose answer is the same every run should not consume a wakeup. Concretely, more than forty percent of our scheduled jobs run as plain scripts; that accounting lives in another article, so here only the effect on the bill: it holds the call-count multiplier still.

Fourth, separate the calls that need reasoning from those that do not. Reasoning is 44.3 percent of our output and we have never tuned it, because I lack a test that can tell which calls genuinely require deliberation. Touching it without that test means tuning by feel, and the money saved comes back as judgment errors. The item is on this list to mark a boundary: not every place you could save is a place you should save this quarter.

Our position

The unit of agent cost is a product of three quantities: fresh bytes that must change on every wakeup, wakeup count, and price per token. Architecture fixes two of the three. Procurement fixes the third. That is why swapping models rarely solves the problem, and why waiting for cheaper tokens is not a strategy on agent workloads.

It also yields a testable prediction. Any agent system that resends its context every turn will see its bill trend toward the square of its task history, regardless of token price. The inverse is the more useful form: if a platform’s customers show bill growth persistently below the growth of their task histories, that platform either stabilizes the prefix or moved state outside the context. That is a better way to evaluate a vendor than a benchmark score.

Our own choice is to move state out. Files live on disk, conclusions live in memos, indexes live in a local database, and each turn carries only what this decision needs. The cost is that every turn requires an explicit decision about what to include, which is more work than dumping everything in. The benefit is that the ceiling on the bill is set by our design rather than by how long a task has been running.

Three questions for anyone building a budget, all answerable on the spot. Per completed task, how many input tokens are rereads and how many are new information? What is the cache hit rate, and will the platform show it to you? After context compression, has the loss of conclusions ever been measured? Get those three numbers and your bill becomes predictable. Miss them and next year’s price change leaves you with nothing to negotiate against, because you cannot say which part moved.

How the numbers were counted

Every token figure comes from this machine’s Hermes state store, table session_model_usage, aggregated from the first record in May 2026 through 28 October 2026 09:01: 10,967 usage rows covering 189,417 API calls. New input 2,944,657,059; cache reads 10,425,268,352 counted separately; output 155,340,340 including 68,754,534 reasoning tokens. The cost columns in this table carry status unknown, and we have no trustworthy dollar price, so the article reports token quantities only and never converts them to money. The 3.7 GB figure is the main state.db file, with 540,468 messages, 11,166 sessions and 19,577 compacted messages. Every number is recomputable on the same machine.

References

Anthropic prompt caching documentation, cached input billed below the standard input rate (vendor documentation, primary): docs.anthropic.com/en/docs/build-with-claude/prompt-caching

Microsoft Azure blog, context billed every turn and agent context cost often the largest expense (vendor blog, technical explanation): azure.microsoft.com/en-us/blog/the-economics-of-agent-optimization-context-engineering-for-enterprise-ai-agents

Cockroach Labs blog, relaying Gartner’s projection of a roughly 90 percent drop in trillion-parameter inference cost by 2030, plus three reasons bills will not follow (vendor blog relaying a third-party forecast): cockroachlabs.com/blog/agentic-ai-costs-at-scale

Sourcegraph engineering blog, quadratic cost in sequence length and lower recall from long context (vendor technical blog, reported experience): sourcegraph.com/blog/context-engineering

Call counts, token composition, cache share, message and compaction counts are first-party queries against this machine’s state store, not relayed.