Twelve steps of latency: where the seconds go

A twelve-step agent task, one model call and one tool call per step, with a person watching the screen. In a single region the model answers in 200 milliseconds and the tool in 50. Multiply 250 milliseconds by twelve and the task takes three seconds.

Nothing exceptional is happening in that number. No step is retried, no tool is stuck, no request queues. It is the ordinary value of an ordinary task. Set it beside Akamai’s State of AI Inference 2026 and the three seconds acquire an origin.

The survey reports four figures. 64% of organisations require sub-250 ms responses for their top use cases. Half of production inference deployments cannot hold their latency at scale. Half of the teams name peak-load latency as their tightest scaling constraint. And 46% still run inference in a single cloud region, while 60% agree that inference should run closer to end users. The report calls the result a latency wall created by centralised architecture.

Those look like four findings. They are three addends in one sum. The 250 ms figure sets the budget for each step, the single region sets the length of each round trip, and the peak sets the queueing term. Three seconds is what they add up to. It is not the consequence of choosing the wrong GPU.

1. Break the latency all the way down

End-to-end time is a sum: the number of serial steps multiplied by the model time, network round trip and tool wait for each step, plus the retry count multiplied by step time, plus the initial queue.

Every term divides further. Model time splits into reading the input, which grows linearly with prompt length, and writing the output, which grows with the number of generated tokens. The network adds at least two round trips per step, one taking the request out and one bringing the answer back, and its length follows physical distance. Tool waits land on someone else’s system. You do not control them, and they are frequently larger than model time.

The cost side reads the same way. We have previously calculated inference bills where system prompts made up 70% of input tokens and only 28% of calls hit the cache. The same repeated context is a line item in the cost table and a stretch of input reading in the latency table. Every time it is read again, it costs time as well as money.

The regional term is the most under-estimated. Put the model call in another region and add a 200 ms round trip per call. Twelve steps add 2.4 seconds, and the task moves from three seconds to 5.4. Put the tool call on the far side as well and it reaches 7.8 seconds. The model still answers in 200 ms each time. What got slower is the round trip, and the number of round trips equals the number of steps.

2. Why the peak term is not linear

Queueing has a rough relationship that is good enough for planning. At 50% utilisation the average wait is about one service time. At 70% it is roughly 2.3 times, at 85% about 5.7 times, at 95% about 19 times. With a 200 ms step, pushing utilisation from 50% to 85% moves the wait from 200 ms to 1.13 seconds.

This is the mechanism behind the observation that extra GPUs do not repair tail latency. Capacity is planned against average concurrency, while latency fails at peak concurrency, and an exponential curve sits between the two. Buying more cards shifts the curve right. It cannot push through, because the denominator of queue time is one minus utilisation, not the number of GPUs.

Half of the teams in the survey cannot hold latency at scale, and this is why. It is not a selection mistake. It is a mismatch of two measurements: procurement sizes capacity against average requests per second, while acceptance looks at p95 latency, and users absorb the difference.

3. Where step counts come from

The design diagram usually has three or four boxes. Understand the request, look something up, act, reply. Production rarely stops at that number.

Most of the surplus has a legitimate reason. Parsing intent takes a call. A retrieval that returns nothing requires a rewritten query and a second attempt. A malformed tool argument requires a retry. A result needs validation, and a failed check starts the step again. A final call turns the trace into an explanation a person can read. None of these appear in the diagram. Each one added during iteration costs another model round trip and another tool wait.

Production step counts therefore run two to three times the design estimate. This is not a discipline problem. It is the shape of the form itself: a single call has been replaced by a chain of conditional calls, and the conditions only appear at runtime.

4. Three layers, in a fixed order

The decomposition above leaves three levers, and the order matters.

First, cut steps. Twelve to six takes the task from three seconds to 1.5, a straight halving. The moves are to parallelise independent steps, replace fixed decisions with rules, and merge repeated retrieval. This layer changes no contract, no vendor and no budget. It does require someone to read an actual execution path, which most teams have never done.

Second, shrink the context. Input reading is a linear term, so dropping a system prompt from 30,000 tokens to 8,000 cuts time to first token. This usually happens alongside the first layer, because long context is often a by-product of many steps.

Third, move inference closer. This addresses the round trip: at 20 ms instead of 200 ms, twelve steps save about two seconds. It is the only one of the three layers that costs money, touches the contract and changes operations, which is why it belongs last. Most teams do it first, because it is the only layer they can outsource.

5. When not to move inference

Three kinds of workload should skip it.

Overnight batch jobs. Nobody is waiting, centralised capacity is cheaper, and latency is not a constraint at all. A multi-region deployment buys a metric with no consumer.

Environments with data residency rules. Financial services, healthcare and public sector work often require data to stay inside a country or province, which means part of that 46% is compliance rather than oversight. In those settings only the first two layers are available, and forcing proximity trades milliseconds for a regulatory problem.

Multi-region setups that need strict consistency. Model versions, caches, indexes and feature stores have to stay in step across regions, and version drift produces behavioural differences that are harder to explain than the latency it removed. Two seconds saved can buy an inconsistency nobody can reproduce.

The test is one question. Is a person waiting on this task. If yes, proximity is worth costing. If not, skip it.

6. Per-step timing, recorded in the trace

None of this arithmetic is usable without measurement. A workable trace records three numbers per step, covering model time, network round trip and tool wait, plus one total for the task.

Without the network term you can never tell whether the vendor is slow or the data centre is far away. Those two findings lead to opposite actions, one to a different model or vendor, the other to a different placement. Most teams only hold a total duration, so they guess between the two.

The same fields become the acceptance test. If the measurement point written into the contract does not map onto a field in your trace, you will discover on acceptance day that both sides were measuring different things, and neither was wrong.

7. Four contract terms worth changing

Once the decision to buy is made, four terms matter more than the SLA percentage.

One, the measurement point. An SLA normally covers server processing time and leaves out your network leg and your tool calls. Ask the vendor to state where the clock stops: first token, full response, or end to end. The gap between those three readings can be a factor of several.

Two, the region list and the right to route. Whether a class of calls can be routed elsewhere at peak, who decides, and whether the data may cross. All three questions belong before signature, not on the day capacity runs out.

Three, the degradation policy. Fall back to a smaller model, serve from cache, or queue. Choose one and write it down. Without a clause, the default behaviour at peak tends to be the worst of the three.

Four, the unit for rate limits. An API that limits requests per second will throttle an agent in ways nobody expects, because one task looks like twelve requests from its side. Per task and per request are two different availability regimes.

8. Our judgment

First, the unit of latency has changed. Once the unit is a task rather than a call, call-level optimisation hits a ceiling. Shaving a call from 200 ms to 180 ms saves 240 ms across twelve steps. Cutting twelve steps to six saves 1.5 seconds. The two differ by an order of magnitude, and most optimisation lists are still written in the first unit.

Second, proximity is a structural consequence of agent architecture rather than an extension of content delivery. Once a task has a dozen steps with two round trips each, physical distance becomes the most overlooked and most quantifiable term in the sum. It is also the only term a vendor can take off your hands, which is why vendors emphasise it.

Third, the numbers are usable and the conclusion has to be recomputed against your own tasks. The 50% figure describes how common the problem is, not whether one of your workflows has it. Split into measurement point, regions, degradation and rate limits, the report is a requirements list. As a buying rationale it is worth much less.

Fourth, the most expensive item in this batch does not cost money. Step counts are written by your own engineers and context is assembled by your own code. Both can be reduced with time rather than budget, and both are usually deferred in favour of the third layer.

9. Three things to do this quarter

Pick one task a person waits on, capture a full trace, and time it by step. Most teams find on the first attempt that waiting occupies more of the task than generation does.

List the steps and mark which can run in parallel, which can become rules, and which can merge. Set the first target at halving the count. Nobody needs to approve this one.

Then take a real task to the vendor and ask whether it can account for every segment of those three seconds. If it cannot, put the measurement point into the acceptance criteria before discussing price.

References

Akamai, The State of AI Inference 2026: akamai.com/lp/the-state-of-ai-inference. The 64% sub-250 ms requirement, the 50% figure on production deployments losing latency at scale, the 50% naming peak-load latency as their top scaling constraint, the 46% tied to a single cloud region, the 60% favouring proximity, and the latency wall wording all come from this page. The report is vendor research whose conclusion matches Akamai’s distributed inference products, and it is labelled as such here.

DigitalOcean, 12 Best Inference Providers for AI Agents in 2026: digitalocean.com/resources/articles/best-inference-providers-ai-agents. Its citation of the Currents report puts inference cost as the top barrier to scaling for 49% of respondents, also vendor survey data.

The per-step budget, the 200 ms cross-region round trip and the queueing multiples are calculations from published latency ranges and standard queueing relationships, not measured figures from any vendor. Step counts and timings in the examples are illustrative, used to show the structure of the sum rather than the performance of a specific system.