Topic Cluster
AI Infrastructure
Model capabilities have converged; infrastructure is where differentiation happens. Engineering decisions on inference cost, model routing, vector databases, RAG pipelines, and agent runtimes determine the marginal cost and latency of AI applications. This topic dissects infrastructure choices with measured data: how much, how fast, how reliable.
26 articles
Agent traffic 4.7x: analytics saw nothing
DataDome counted 25bn agent requests in Q3, up 41.5%. Agentic browsers ran 4.7x in September, led by a newcomer that declared nothing at all.
Inference vendor risk is now counterparty risk
Gartner sees $3.5tn of AI infrastructure exposed to impairment by 2030 and 80% of neoclouds gone: your inference vendor is now a counterparty with a horizon.
AI chip financing at 11% while chip stocks slide
A $5bn chip loan left $3.7bn ununderwritten at about 11%, while Korean chip stocks fell 7% in a day. Two prices for the same assets.
Inference tier: why RTX Pro 6000D is bought
Export controls draw the line at a capability tier, not a border, so 1 million-plus cards below it get bought as compute.
Agent Workflow Costs Rise as Token Prices Fall
Token prices fell 280-fold in two years, yet enterprise AI bills keep climbing. The billing unit moved from one answer to a task that runs a dozen tool calls.
Prompt cache: 9.5x the volume, 19% of the cost
Over 30 days: 389M fresh input tokens, 3.70B cache reads. Cached volume ran 9.5x the fresh input at 19% of its cost; a lower rate does not move it.
Agent billing: per turn, per wait, per active CPU
3 vendors shipped agent platforms in one week with different meters: per turn, per wait, per trace. The meter decides who pays for waits and retries.
AI agent monitoring: a 1-token ping misses a 429
Quota is metered in tokens; the health check is not. A one-token ping returns 200 while a 9,000-token job gets a 429.
Long context vs RAG: cache hit rate decides
Cache hit rate, not window size, decides whether resident context beats retrieval. The crossover sits near 20 reuses per cached block; 3 architectures compared.
AI agent timeout: failures are bimodal, not slow
Of 423 failed tool calls in our log, 53% errored within 2 seconds, 16% sat at the 170s timeout cap, 31% fell between. The cap is a classifier, not a knob.
Split capacity risk from price risk in AI buying
Alibaba's CEO says long-term demand far outpaces supply, capped by global supply chains. Tencent priced 2K images at ¥0.15. Capacity and price are separate.
Why cost per successful task beats token price
Token prices fell 41% since March, yet per-employee AI spend at the heaviest users dropped 10%. Measure cost per successful task, not token price.
AI capex: the $300B obligation off the books
Per FT, ~$300B of AI infrastructure exposure sits off balance sheet via guarantees and SPVs; Alphabet's lease guarantees hit $43.8B with under 2% on-book.
AI memory cost: the upstream of model pricing
TrendForce: 2Q26 DRAM revenue +59.5% QoQ to $154.7B, supply still lagging demand; CXMT's 5th-gen node adds a source. Memory sets 2027 AI cost.
Model buying is two markets: volume vs money
Open-weight models run 56% of tokens, 14% of spend; 9/10 teams stepped Fable 5 down to Opus 5. Our judgment: buy by task tier, not average token price.
Routing savings aren't bought: ACL’26 benchmark
ACL’26 LLMRouterBench finds commercial routers fail to beat a simple baseline (arXiv 2601.07206). Our take: deterministic policy beats buying a router.
Inference outspends training. Buy switching power
Gartner: 2026 inference spend ($23.3B) beats training ($19B); agentic costs >5x by 2028. Cost = configuration, not model price. Buy switching, not tokens.
Agentic open weights: verify, don’t believe
Smaug's 15-20% gains and 10-100x savings are vendor claims. Open weights make them checkable: evaluate on your own traces, measure cost per completed task.
Context layout beats model price on agent cost
2026 evidence: caching cut agent costs 41-80%; a 7%→84% hit-rate refactor saved 59%; 4-min keepalives beat 30-s pings. Hit rate, not token price, drives cost.
The token price war changed the real cost lever
OpenAI cut Luna 80% to $0.20/M; Sonnet 5 stays $2/$10. Our view: the price war ended model choice as the cost lever — cache, batch and retries set the bill now.
Agent cost control moves to execution layer
Agent cost control moves to execution layer: budget pre-enforcement, per-call routing, full metering. Sapiom's $35M: dashboards audit, execution layers govern.
Model routing is not a config: the closed loop is
RouteNLP pilot (ACL 2026): -58% inference cost, p99 1,847ms to 387ms. Our take: static routing gets half the savings; the closed loop is the other half.
NVIDIA buys HF: whose boundary holds the audit trail?
NVIDIA's $12.9B HF deal ships audits 'below the security boundary' — all inside its own wall. Vertical integration moves the trust gap; it doesn't close it.
Stateless MCP: payback is deleted infrastructure
MCP's 2026-07-28 spec: stateless core — no handshake, no session store, tracing built in. First upgrade paid in deleted infrastructure, not added features.
Do you need a vector database? RAG in 2026
pgvector beats split vector stacks: 92% lower filtered latency, 74% tenant-scoped, zero sync gap. Decide by scale, freshness, query shape, permissions, ops.
LLM routing: conditions behind 40-80% savings
Route each request to the cheapest capable model. Behind 40-80% savings claims: cheap share past 50%, an eval gate, closed-loop distillation.