Context layout beats model price on agent cost

If your agent bill is growing, do not switch models first. Two 2026 empirical studies converge on the same conclusion: the first cost lever for agentic workloads is not token unit price — it is context layout and cache lifecycle. Same model, same prompts, different content order: the bill can differ by more than 2x. Pricing tables decide the unit price; context design decides how many units you pay.

Evidence 1: caching rescues agent loads — but only at a healthy hit rate

Prompt caching reuses computed key-value tensors: requests sharing a recent prefix bill at roughly 0.10x input price — up to ~90% off on Anthropic and OpenAI’s newer models, ~75% on Google’s implicit cache, ~0.1x on DeepSeek with no storage fee (2026 vendor list prices). Because the model recomputes nothing it has seen, output is byte-identical: this is the only free-and-lossless cost lever in the stack. The PwC team’s study “Don’t Break the Cache” (arXiv 2601.06007, Feb 2026) measured across 500+ agent sessions with ~10,000-token system prompts: API cost down 41–80%, time-to-first-token improved 13–31%. The same study carries the counter-intuitive warning: naive full-context caching can paradoxically raise latency in some configurations. Caching is a design discipline, not a toggle.

The dividing line is hit rate. Anthropic’s 5-minute tier charges a +25% write premium; breakeven is roughly 1.4 reads per cached write, and below ~30% hit rate the premium costs more than the reads save. On stable-prefix workloads, a hit rate stuck under 60% is a structural problem, not bad luck (figures per the Digital Applied guide, Jun 2026, linked below).

Evidence 2: ProjectDiscovery’s 7% → 84%: move working memory out of the system prompt

The most instructive case is ProjectDiscovery: after relocating dynamic working memory out of the system prompt into a user message at the end, cache hit rate went from 7% to 84% and overall LLM cost fell 59%, with 9.8B tokens served from cache. The mechanism is unforgiving — the cache is keyed on the literal token sequence, so a single changed character before the cache boundary invalidates that block and everything after it. Order prompts most-to-least stable: tool definitions, system prompt, reference docs, conversation history, then the live query.

Evidence 3: pauses are the #1 cache killer; keepalives are the antidote

An agent loop is think–act–wait: it sends a request, then runs a build, a test suite, or waits for human approval for minutes, and only then sends the follow-up that would have reused the cached prefix. Provider caches expire in minutes — Anthropic’s default TTL is 5 minutes, OpenAI clears after 5–10 minutes idle — so the follow-up pays full prefill again. arXiv 2607.19214 “Keeping the Cache Warm Pays” (Jul 2026) quantifies the fix: a 30-second keepalive held 40/40 prefixes warm through a 600-second gap (no-keepalive baseline: 0/48), cutting post-pause request cost up to 12.5x. And keepalive cost falls monotonically with interval, so the economical choice is the largest interval safely under the TTL: ~4 minutes at Anthropic’s 5-minute tier, an 8x reduction in keepalive spend versus the 30-second convention.

The same study delivers a second counter-intuitive result: DeepSeek’s re-prefill is too cheap to insure and Google’s cache almost never evicts — keepalives buy latency only on those two, not savings. Measured eviction points: Anthropic ~5 min, DeepSeek ~10 min, OpenAI ~30 min, Google never. Your provider determines whether a keepalive strategy even makes sense.

Our judgment: cache hit rate is the agent cost KPI

Industry discussion of agent cost still spends 90% of its attention on model selection and price per token — a legacy of last cycle’s token price war. The 2026 bill structure has changed: input tokens routinely dwarf output, and most of that input is the same context re-sent over and over. The three variables that actually decide the bill are: how much of the prefix is static, where the dynamic content lives, and whether the cache is still alive when the agent returns from a pause. None of them are model properties. They are all context design.

We validate this on our own architecture. OOMeta’s agents run on deepseek-v4-flash (automatic prefix caching, no storage fee) and gemini-3.5-flash; long sessions with skills (SKILL.md) loaded on demand form a naturally stable, cacheable prefix, and our “LLM proposes, scripts persist” principle keeps state on disk instead of in context — controlling prefix bloat at the source. A daily cron fleet re-executes near-identical system prompts, which is cache-friendly by construction: hit rate is the metric we watch first. The flip side: every high-value flow that requires human approval pauses for minutes, and keepalives decide whether that segment bills at 0.1x or full price.

Playbook: four moves you can make today

Act: answer one question first

Decision question for AI app leads: what is your cache hit rate? If the answer is “we have never looked,” 41–80% of your cost space is probably sitting on the table. Wire hit rate into your cost dashboard, run the four moves above, and compare the bill in two weeks. We bet you land on the same conclusion we did: the first cost variable for agents is not the model — it is the context.

References: arXiv 2601.06007 “Don’t Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks” (PwC team, Feb 3, 2026, independent research) https://arxiv.org/abs/2601.06007 · arXiv 2607.19214 “Keeping the Cache Warm Pays: Keepalive Economics for Agentic Workloads” (Jul 24, 2026, independent research) https://arxiv.org/abs/2607.19214 · Digital Applied “Prompt Caching in 2026: Cut LLM Costs, Keep Quality” (Jun 16, 2026; includes the ProjectDiscovery case and provider pricing) https://www.digitalapplied.com/blog/prompt-caching-2026-cut-llm-costs-engineering-guide · Anthropic Prompt Caching docs (write premiums / TTL tiers, vendor-reported) https://platform.claude.com/docs/en/build-with-claude/prompt-caching · OpenAI Prompt Caching docs (vendor-reported) https://platform.openai.com/docs/guides/prompt-caching