Inference cost per workflow is the paradox

A coding agent trace was taken apart and costed line by line: 996,500 tokens in total, of which 940,000 were cache reads, 1,200 were uncached input, and 5,300 were output the model actually generated. In other words, 94.5 per cent of the money went on re-reading the same context, while the code it produced accounted for less than half a per cent. Augment Code published that trace inside a cost analysis. It is one sample, but it shows the shape of an agent invoice very clearly.

The bill grows with the process, not with the answer

Here is why it forms that way. Every time an agent takes a step and calls a tool, it resends everything accumulated so far: conversation history, system prompt, tool definitions, the results of the previous call. On a twenty-step task, the same opening block is read twenty times. Output appears only at the end, while input repeats at every step. So what an agent costs tracks how long it behaved, not how much it wrote.

That is also why a budget built on per-token prices stops matching the invoice about three months after an agent ships. The person setting the budget reads a price list, and a price list contains prices. The finance team receives a bill, and a bill measures volume. Neither party is wrong; they are measuring different objects.

Caching is not free either. Writing a prefix into cache is billed at a higher rate, and only the read is cheap. So the question is not whether a prefix can be cached but how many times it will be reused. On a twenty-step task, a system prompt read twenty times almost always pays for itself. On a three-step task, often not. Run that arithmetic on your own traces; somebody else’s sample can only calibrate the direction.

Unit prices fall, invoices climb

Gartner named this on 17 August. It predicts that inference cost per agentic workflow will more than quintuple by 2028 while the price of a single token keeps falling. Gartner calls the combination the inference paradox: better unit economics escalating the overall cost of AI, with no clear path to commensurate and predictable value.

A second forecast from the same firm draws the opposite curve: by 2030, inference on a one-trillion-parameter model will cost more than 90 per cent less than it did in 2025. The two curves do not conflict, because they measure different things. One measures how many tokens a workflow burns to finish; the other measures what a token sells for. Cheaper material, faster-growing consumption, and consumption sets the total.

The mechanism Will Sommer of Gartner describes is plain. A chatbot reads a question and quickly returns a probabilistically reasonable answer; an agent has to keep reasoning, negotiating and questioning itself. Routing a task to an agentic reasoning model raises the provider’s inference cost by at least five times compared with a chatbot interaction, and more as the task grows. A Gartner analysis in March put the range at five to thirty times.

For procurement this is counter-intuitive. You think you are comparing prices when you are actually comparing consumption, and consumption is set by your architecture rather than by anyone’s rate card. Sommer’s wording is that defaulting to generic autonomous intelligence will produce costs orders of magnitude higher than an optimised product ecosystem. That comes from a firm paid for forecasts, so it is not necessarily your bill, but the mechanism is worth keeping.

Seventy per cent of input is system prompt, and 28 per cent of calls hit cache

Datadog’s State of AI Engineering 2026 slices the same problem with LLM telemetry from more than a thousand customers. It found that 69 per cent of all input tokens in customer traces were system prompts: internal instructions, policy definitions and tool guidance passed down the chain. That is the shape of a heavily scaffolded agent, where guardrails and tool descriptions appear almost verbatim in every single call.

Inside the same report sits its counterpart. Even among models that support prompt caching, only 28 per cent of LLM call spans showed any cached-read input tokens. More than seven calls in ten reprocessed the full prompt. Set 69 against 28 and the waste becomes visible: the largest block of cost is also the block most amenable to caching, and most of the time it is not cached.

The report points at prompt layout. If dynamic content is injected too early, or a block that should be stable gets reordered or rewritten between requests, prefix reuse breaks and the cache cannot hit. That is an engineering detail, and it sets the size of the bill. Fixing it requires neither a new model nor a rewrite of business logic, only keeping the opening block unchanged.

Two more numbers show the pool growing. Median tokens per request doubled year over year among Datadog customers, and the ninetieth percentile quadrupled. At the same time more than 70 per cent of organisations run three or more models, and the share running more than six nearly doubled. More models, longer context, and more of the repeating part.

Context cannot be solved by deletion

Confronted with 69 per cent, the first instinct is to shorten the system prompt. The direction is right and the floor is low. Guardrails, policies and tool guidance are long because they carry reliability. Delete them and the agent pays the cost back another way: more retries, more ambiguity, more human review. The useful question is not what can be removed but which part must stay identical on every call.

Fireworks published retry figures by model. On a ten-step task running against a model with an 18.6 per cent retry rate, the average run wastes 2.3 attempts, roughly 36,500 tokens and 5.7 seconds of dead time. At ten thousand agent tasks a day, the worst-performing model burns more than forty thousand dollars a year on inference that produces nothing. A model that looks cheaper per token can cost more per outcome, and the inversion only becomes visible once retries enter the cost.

Failure inflates the bill on its own. Datadog recorded 5 per cent of LLM calls returning an error in February 2026, with 60 per cent of those caused by exceeded rate limits. In March the overall error rate fell to 2 per cent, yet rate-limit errors still numbered nearly 8.4 million. An agent that hits a limit retries, retries raise concurrency, concurrency hits the ceiling again, and a local wobble becomes a sustained outage. Every retry is a token that produces nothing.

Change the unit to one accepted outcome

At this point a usable measure follows: cost per accepted outcome rather than cost per million tokens. The numerator is the spend of the whole trajectory, with input, cache reads, cache writes, reasoning and visible output all included. The denominator is the number of tasks that passed an automated check, not the number launched.

The two denominators diverge sharply. Tasks launched hide retries and failures; tasks passed bring the apparently cheap model back to its real shape. A model that costs ten per cent more per token and gets the task right first time usually beats a cheaper one that keeps patching. That arithmetic rarely reaches a procurement meeting, because it demands a finer grain of logging than a rate card requires.

Three things have to be in place, and none of them is optional. First, every request carries a task identifier, so spend can be grouped by trajectory; aggregating by key or by day will never tell you which class of task is growing. Second, the prompt prefix stays stable, policies and tool descriptions fixed at the front with dynamic content behind them, and cache hit rate is treated as a first-class metric sitting next to latency and error rate. Third, each task gets a budget ceiling and a termination condition, enforced during the run rather than discovered at month end.

One more layer is routing. Datadog observes leading teams running inference like a pipeline: light models for extraction and tagging, frontier models for synthesis and judgement, with each stage evaluated on quality, cost and latency. That is not a cost trick; it is an admission that different steps carry different value density. Gartner puts it more bluntly, saying there is no reliable and economical one-size-fits-all model on the horizon.

Assemble these and the cost curve has only three knobs worth turning: trajectory length, cache hit rate, and routing tiers. The first two set unit cost; the third sets how much judgement the same money buys. All three live in architecture. None of them lives on a purchase order.

What these numbers cannot support

Three limits come first. The fivefold figure from Gartner is a forecast rather than a measurement, and it addresses the cost structure of AI product vendors, which need not match an enterprise running agents internally. Datadog’s sample is its own customer base, companies with relatively mature infrastructure, so its medians are not an industry census. The five-to-thirty-times range comes from a Gartner analysis in March, relayed through several outlets, with the original report unpublished.

What holds is the direction: agent cost grows with trajectory length, falling unit prices do not offset it, repeated context is the largest block in the bill, and cache utilisation sits well below what the mechanism allows. Any specific multiple should travel with its source and year, including the share we computed from that single trace.

Our judgement

Our judgement is that agent cost is an architecture problem rather than a procurement problem. Unit price is a curve you do not control; consumption is a curve you do. Most budget models only contain the first one. When the bill doubles in month three, the usual explanation is not a price increase but a longer trajectory.

Three moves follow. This week, switch cost attribution from per vendor to per task and rerun the historical data, then look at the most expensive ten per cent of tasks; they tend to cluster in a few places where the task was left too open. Second, plot cache hit rate alongside the share of input tokens that are system prompt, because the two together show immediately whether the prefix is being broken. Third, give every task a budget ceiling and a stopping condition, enforced as the run proceeds, because a runaway loop does not stop by itself.

The question worth leaving with a buyer: can you say what your agents spent last quarter for each accepted outcome they delivered? If that number cannot be produced, then the budget you approved and the invoice finance received are measuring two different things.

References

Gartner press release, “Gartner Predicts AI Inference Costs Per Agentic Workflow Will Increase More Than Fivefold Through 2028”, 17 August 2026 — source for the inference paradox, the fivefold figure, the five-times floor against chatbot interactions and Will Sommer’s quotes

Datadog, “State of AI Engineering 2026”, telemetry through March 2026 — source for 69% system-prompt share, 28% cache-read calls, doubled median tokens, quadrupled p90, 70% running three or more models, 5% and 2% error rates, 60% rate-limit share and 8.4 million rate-limit errors

Augment Code, “AI Coding Cost Analysis: Where Token Spend Really Goes in an Agent Loop” — source for the published trace with 940,000 cache reads, 50,000 cache writes, 5,300 output and 1,200 uncached input tokens

Fireworks AI, “Agent Execution Tax” — source for an 18.6% retry rate producing about 2.3 wasted attempts, roughly 36,500 wasted tokens and 5.7 seconds, and more than $40,000 a year of wasted inference at ten thousand tasks a day

CIO Dive, “AI inference costs set to plunge: Gartner” — source for the 90 per cent decline by 2030 on a one-trillion-parameter model and the five-to-thirty-times token statement

DoiT, “Cost Per Task: Claude vs OpenAI LLM Economics 2026” — method for logging tokens per trajectory, recording checked outcomes and computing success-adjusted unit cost