Why agents fail in production:
the SOP-Bench lesson

Your agent shines in the demo — so why does it fall apart in production? On August 21, Amazon presented SOP-Bench at KDD 2026, an open benchmark that evaluates agents on real business procedures. It delivered three results that should worry any enterprise: upgrading the model can lower success rates, adding tools can make an agent worse, and no single model-plus-agent combination excels across all procedures.

Existing benchmarks test capability, not procedures

The problem with most agent benchmarks is that they test isolated capabilities with clean, machine-formatted prompts — tool selection, retrieval and reasoning each in their own lane. Real procedures require coordinated multi-tool use, interpreting ambiguity and domain expertise. SOP-Bench is the first benchmark to pair genuine enterprise procedures with functioning tools and ground-truth answers: an agent earns its score by completing the procedure, not by producing text an automated grader happens to like.

It covers 12 business areas — healthcare intake, dangerous-goods classification, customer service, content moderation, financial compliance, warehouse inspection and more — with over 2,000 tasks. Each task ships with the tool interfaces an agent requires and a correct outcome. The framework keeps a full record of every tool call and reasoning step, grades outcomes against ground truth, and lets failures be traced back to the step where they happened. SOP-Bench is a framework, not a fixed question bank: teams can drop in their own agents or extend it to new domains with the same method.

Upgrading the model can lower success

The most surprising finding: upgrading the model sometimes lowers performance. On the reasoning-style agent, the newer Claude 4.5 family scored lower than the older Claude 4 family; the same reversal held when comparing individual models on the same setup. For production teams this is the finding that matters most — a routine upgrade can silently lower your success rate with no obvious signal that anything changed, and the only reliable way to catch it is to test on the procedures you actually run.

More tools, worse agent

SOP-Bench ran an experiment that goes straight at a common pain point. Take a video-annotation procedure and give the agent two versions of its toolkit: one holding exactly the six tools the task required, the other keeping those six but burying them among 20 plausible-looking tools that did nothing useful. Success nearly halved with the larger toolkit — even though every tool the agent needed was available. The lesson is that capability is not free, and trimming an agent's tools to fit the task may be a key component of getting it ready to deploy.

No combination wins everywhere

After 11 frontier models and two agent designs (function-calling and reasoning) ran the 12 procedures, no single model-plus-agent combination came out ahead across the board — the pairing that performed best on one procedure was often a weak choice on another. The gap between procedures was wide: on the easiest, triaging incoming emails by intent, agents were correct about nine times out of ten; on the hardest, annotating objects in a driving video, about one in four — a more-than-threefold gap across the suite.

The reasoning agent came out slightly ahead on average, yet it won on only eight of the thirteen procedure runs and took about a third longer per task. The shape of the procedure, not a single overall average, should drive the agent choice. Trusting one benchmark score would tell a team almost nothing about how the same setup would behave on the next use case.

Reproducible scoring, traceable failure

Why should enterprises take this evaluation approach seriously? Because it turns evaluation from folklore into engineering: each procedure is just four things — the SOP text, the tools an agent can call, the tool specifications, and a set of test cases with known answers. Scores are reproducible and failures are localizable. A team can run its own SOPs first, before trusting an agent in production. For an agent deployed alongside human operators on procedural work, a static skills test is not enough — its real weaknesses surface only in sustained, tool-using runs against realistic procedures.

Three actions for your team

SOP-Bench's results are not a verdict on any single model. They are a map: raw capability does not guarantee reliability on the procedural work businesses depend on. For enterprises pushing agents toward production, the map points at one discipline — before you deploy, test on real procedures with real tools and real answers. It is cheap, and it catches the failures that demos never show.

References

• Amazon Science: SOP-Bench: A new benchmark for evaluating AI agents on real business procedures (2026-08-21) — https://www.amazon.science/blog/sop-bench-a-new-benchmark-for-evaluating-ai-agents-on-real-business-procedures
• SOP-Bench release on GitHub and HuggingFace (12 procedures, generated tools, baseline agents, evaluation code) — linked from the Amazon Science post above