AI agent simulation testing
4.8x, not proof

On one digital bank’s card-servicing line, the same agent took 212 days to move from its first version to its fourth — 21.2 days per version. What held it back was not writing code. It was verification: the only way to learn whether a change helped was to wait for real customers to have the same conversation again, and real customers cannot be scheduled, replayed or experimented on.

On the company’s next product line, verification became a simulation gate that runs before release. Five versions shipped inside 22 days, 4.4 days each. Same team, same class of task, a 4.8x difference in version cadence.

The numbers come from a paper submitted on 24 September 2026, Screen Before You Serve, by Nubank, which deployed the system, and Snowglobe, the simulator vendor. That makes it jointly self-reported, with no third-party audit: it can be written as “Nubank reports”, never as independently verified and never as an industry benchmark. What follows separates what this method can prove from what it cannot.

One: 21 days per version is a verification problem

An agent has exactly one source of truth, and that is production. Production also has two properties that disqualify it as an everyday test bench. First, the distribution of real conversations is outside your control: today it is all balance enquiries, tomorrow it is refund disputes, and the same test set will not reproduce. Second, the cost of a mistake is paid by a customer, so a bad routing decision or a wrong answer does not show up as a failed log line but as a service incident.

Together these push teams into batching. Several changes ship as one release to limit exposure, and when something breaks the whole thing is rolled back, which turns root-cause work into archaeology. Versions get coarser, ownership gets blurrier, and the room for experiments shrinks. Twenty-one days per version is not an engineering capability problem. It is the arithmetic of insufficient verification supply.

Two clocks are being confused here. One runs from writing code to being able to ship, and it responds to refactoring, CI and better tooling. The other runs from having a change to being willing to ship it, and it is set by how fast you can learn whether that change is good. The first is engineering throughput; the second is information throughput. Most teams invest in the first while the second is what stalls them.

Two: replace the production backend with synthetic results

The mechanism is not complicated. Synthetic customers hold multi-turn conversations with the real agent, but every tool call the agent makes is intercepted and answered with a scenario-specific synthetic result instead of reaching a production backend. The simulator needs two inputs: a description of the agent and a definition of its tools. The team used 8,000 real conversations, split evenly into development and test sets, to profile the simulation and check its fidelity.

The design decision that matters is where the interception sits: at the tool boundary, not at the model output. That location defines what the gate can test — multi-turn decisions, tool selection, argument construction, routing logic, all of which live around tool calls. It also defines what it cannot test, which is the subject of section four.

Speed differs by orders of magnitude. A hundred trajectories finish in under ten minutes, where replaying the same sample against live traffic takes hours or days. Inside simulation the evaluator caught failed tool calls and a serving configuration missing its tool-call parser — the kind of defect that otherwise surfaces only in production, where attribution is expensive.

Fidelity has a number attached: the minimum cosine distance between simulated and real conversations is 0.035, and across four deployed versions simulated scores correlate strongly with production binary scores. Both statements support comparability, not equivalence. They are the reason to trust simulation output, and the boundary that tells you when it stops applying.

Three: the live A/B numbers, with their denominators

The claim that the process changed production outcomes has A/B backing. Moving from the card delivery line to card management, self-service resolution rose 4.90 percentage points (interval 4.08 to 5.71, n=27,800) and transactional net promoter score rose 36.69 points (interval 32.81 to 40.57, n=2,000). Both are Nubank’s own reported measurements.

A second step was model selection. The team screened open-weight models with over 16,000 simulated conversations and settled on a 122B Qwen3.5 model with reasoning enabled. The following live A/B showed self-service resolution up 8.82 points (interval 7.95 to 9.69, n=8,400) and p95 latency down 25%, while transactional NPS moved −1.21 (interval −3.97 to 1.55), which is not statistically significant.

That last figure deserves to stay in the story. It says the latency improvement did not translate into customer sentiment: the model changed, the operational metric moved, the experience did not. Delete it and you have a clean scoreboard; keep it and you see that the real change was iteration speed, not the whole of the product experience. Screening happens in simulation because it is cheap and repeatable; acceptance happens in production because it is expensive and scarce. That division of labour is exactly what this contrast shows.

One more figure has to be fenced off. A “20x faster” claim circulates in third-party retellings, and it has no public definition. The paper’s 4.8x is a ratio of version cycle days. They are different measures, so they cannot be used interchangeably, and the larger one should not be promoted into marketing copy.

Four: a gate proves eligibility, not behaviour

This is our judgement, and it is the point of the article: a simulation gate produces a certificate of eligibility. It shows that a candidate is fit to face real customers. It produces no facts at all about what the candidate does once it is live.

The evidence is the paper’s own list of exclusions: backend behaviour, real latency, persistent state and side effects. Those four are precisely where production incidents come from. An agent that looks flawless in simulation can time out because a downstream interface rate-limits it, can repeat an action because state did not persist between turns, can double an operation because a retry policy stacked on a side effect. None of that appears in synthetic results, because synthetic results are designed to be accommodating.

The most common buyer mistake is treating screened as proven. A gate answers whether something may ship. What happens after it ships is answered by a different chain: task-level records outside the agent itself, covering what the task was, what it cost, what the outcome was and who verified it. Without that chain, the renewal conversation a year later still runs on a pre-release simulation report while the budget process is asking post-release questions.

One engineering seam is easy to miss when drafting the plan. The paper notes that when simulated traces diverge from production schema, identifiers or telemetry, export and reconciliation eat the iteration gains. In plain terms: the time you saved in the version cycle comes back as reconciliation hours. The operational requirement is one sentence long — the simulation environment reuses the live MCP schema. It reads like a detail; the economics of the whole method depend on it.

The scope limit should also be stated. This method works best where conversation volume is high, the workload is dialogue-shaped and tool boundaries are clean: customer service, ticket handling, order lookups. Move to side-effect-heavy workloads with long-lived state — fund transfers, inventory decrements, cross-system long transactions — and the covered share shrinks fast, because the hardest part is exactly what simulation omits. Lifting the conclusion into those settings is using a gate as a ledger.

Five: tune inference settings before chasing precision

A controlled comparison in the paper’s appendix cuts against today’s cost-optimisation instinct. Quantising the model to nvfp4, fp8 or bf16 produced score differences of 2.2, 0.8 and 4.1 points respectively, while running six repeats of a fixed configuration produced a standard deviation of 2.2 points. The differences between formats sit inside the noise: time spent tuning quantisation is largely time spent measuring randomness.

What moved the score was the inference setting, worth about 11 points. The mechanism is not cleverer sampling but retrieval: the share of runs that fetched the required company record rose from 16% to 94%.

Translated into engineering order: first confirm the agent gets the records it needs — retrieval hit rate, context assembly, the shape of tool return values. Only then discuss compression, quantisation and bit precision. Doing it the other way round means saving electricity on an agent that never received its documents, and booking the quality difference as noise.

This reading has a testable form. If retrieval really is the mechanism, then in any cost-quality trade-off the retrieval hit rate alone should predict most of the quality movement. The jump from 16% to 94% and the 11 points are two views of one event, not two separate lessons.

Six: three things to ask your team for

First, whether the gate exists and what it has stopped. Over the past 30 days, how many changes were blocked at the simulation stage, and what were the reasons. A gate that has never blocked anything is not a gate; it is a box on a process diagram.

Second, the seam inventory between simulation and production: whether schema, identifiers and telemetry share one source, who owns reconciliation and how often it runs. The value of this list is that it turns “where did the saved time go” into a traceable question.

Third, the second record that starts after release: a task-level ledger carrying task, cost, outcome and verifier, exportable and checkable line by line. Gate plus ledger is the full evidence chain — screened before release, recorded after it. With only the first, you can show you screened diligently. With only the second, you can show the money was worth it. With both, verification becomes an asset instead of a habit.

If only one can be added now, add the ledger. A gate shows whether a candidate was eligible; the budget decision a year later asks what it actually did once it was running.

Sources

Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale: arxiv.org/abs/2609.30137 (submitted 2026-09-24; Nubank and Snowglobe, self-reported).

Joint talk page (third-party retelling, not independent verification): ai.engineer/speakers/shreya-rajpal; simulator vendor page: guardrailsai.com/snowglobe.