OOOMETA/OS
← Back to Insights

November 2026 · 5 min read

The agent observability gap
89% watch, 52% eval

The agent observability gap: 89% watch, 52% eval

Key Definitions

Observability The ability to record what an agent did, including model calls, tool calls, latency and exit points. It answers what happened, not whether it was right.

Eval An executable set of criteria that decides whether one output is acceptable. It lives inside the workflow, so a vendor cannot pre-build it.

Proof of completion The checkable item required to close a task: an artifact path that exists, a reproduction command, or a hash.

Our task queue holds 3,347 tickets and 3,128 of them are closed. Every closure carries an absolute path to an artifact that really exists on disk. If the path does not resolve, the ticket is not closed, however finished it looks.

That rule was not born from neatness. It came from rework. We used to define done the other way: it ran, nothing threw an error, the output read plausibly. That definition fails quietly. The work looks finished, then someone opens the file the next morning and finds an empty document or a version from three days ago. The rework never lands on one person, so nobody can put a number on what it costs.

Out of 1,340 people, nine in ten watch and half can judge

LangChain published a practitioner survey in June with a reading worth comparing against. It collected 1,340 responses between 18 November and 2 December 2025. State of Agent Engineering

Three figures belong in one sentence. 57.3 per cent of respondents have agents running in production. Quality sits at the top of the barrier list for 32 per cent. Close to 89 per cent have observability wired up for their agents, and 52 per cent have evals.

The first number needs care. The same report states that 63 per cent of respondents come from technology, 49 per cent work at companies under one hundred people, and organisations above ten thousand employees are 9 per cent of the sample. What it shows is that technology firms and small teams move fastest. It does not show that a manufacturer with thirty thousand staff is past pilot. Quote it in that room and the buyer will answer with their own numbers, and the argument stops on that one figure while the two rows that matter go unread.

The last number does not have that problem. The thirty-seven points between 89 and 52 do not depend much on company size or industry. They describe something more basic: organisations can see what an agent did, and cannot judge whether it was right. That gap holds for a small startup and widens in a large enterprise, where business rules are more numerous and more scattered.

Sampling and judging are two kinds of labour at two prices

Observability answers what happened: which model call, which tool, how many tokens, how many milliseconds, where the run exited. Those records are produced by sampling. The unit cost is tiny because logging has nothing to do with the business logic. One collector attaches once and serves every agent in the building.

Evals answer a different question: was this one right. There is no universal answer, because right lives inside the workflow. A reconciliation entry posted to the wrong account is a successful tool call in the trace. A contract clause read without the appendix page raises no exception anywhere on the chain. To call it wrong, someone has to say what right means in this business, then reduce that judgement to something a machine can run.

The two cost curves therefore diverge. Sampling rises with call volume, but only in storage and indexing, and slowly. Judging rises with the number of use cases, and most of its cost sits at the definition end, which is human time and does not scale down.

That inference explains why the thirty-seven points persist. A vendor can ship sampling as a switch: install it and data appears, so it sells. Judging cannot be a switch, because the customer has not decided what right means yet, and deciding it for them means defining their business. So the market competes hard on the easy half, and the hard half stays with the customer, year after year.

Why quality outranks cost

One more shift is worth noting in the same survey: concern about cost fell against the previous year while quality stayed first. Price competition pushed inference cost down, the bill stopped hurting, and nobody stepped in to pay for quality.

The reason sits in who pays for a mistake. Inference is an internal cost that shows on a cloud invoice and can be negotiated. Error cost lands in the business, where a support agent calls a customer back, an order is reopened, a report is reissued, and the work falls on named people rather than on any invoice. Of the surveyed organisations, 26.5 per cent name customer service as their primary use case and 24.4 per cent name research and data analysis. Both put output directly in front of a human, and a human pays for each error.

Put an agent in front of a customer and the pricing power over quality moves to the customer. They will not read your trace. They will tell you they were answered wrong. That pressure does not disappear when the model improves. It grows as use cases move closer to the customer.

What 95 per cent accuracy means in production

The arithmetic in this section cites no report. It is multiplication. An agent answers 190 of 200 test cases correctly, which reads as 95 per cent and looks excellent in a demo. Place it at ten thousand calls a day and five hundred of them are wrong daily. If each wrong answer takes a person six minutes to check, correct and reply, that is fifty hours a day, or six full-time people employed by the error rate.

Accuracy is a sampling metric. Production experience is decided by three other things: who the error lands on, how long it takes to surface, and whether it can be replayed once it surfaces. None of the three lives in accuracy. All three live in evals.

This also explains the familiar shape of a beautiful pilot that collapses at scale. At pilot volume errors spread across days and a person absorbs them without noticing. At scale the error density rises, the person absorbing them becomes the bottleneck, and the dashboards stay green because no metric is watching errors. They are all watching runs.

We define done as something another person can check

Back to the opening figures. 3,128 closed out of 3,347 is 93.5 per cent. That rate is not a product of model accuracy. It is a product of how done is defined. Every closed ticket carries three things: the absolute path of the artifact, a command that reproduces the same result, and the acceptance criterion written at the start.

Why a path rather than a score is practical. A score fails quietly. Ninety-two out of a hundred looks acceptable, so it gets accepted and the error travels downstream. A path fails loudly, because the file is not there and the check turns red on the spot. The value of a judgement is not its precision. It is whether a third party can independently reproduce it in a few minutes. That moves evaluation from someone’s impression to something another person can re-execute.

We hit the failure mode ourselves. Our first version allowed artifacts in a temporary directory and accepted a human glance as the check. Then two sessions claimed the same batch, wrote two versions, and each declared itself the only implementation. The fix was not a line in a prompt. It was queue semantics: re-read the pending list before claiming, require a unique completion key, and let a second completion report already closed instead of overwriting.

The same lesson transfers straight to a client eval design. An eval set is not a list of question and answer pairs. It is a set of snapshots of failure: the input, the tool call sequence, the judgement, and the reason a person called it wrong. Fifty such snapshots give you a regression set. Run it before a model upgrade, and the points that drop are the price of that upgrade.

Four buyers, four numbers to move

For the C-suite: the budget case should carry the share of deliveries with a reproducible proof, not the GPU count. That share can be counted from delivery records, so unlike a confidence index it belongs in board material. Moving from one fifth to three fifths in a quarter says more about control than deploying ten more agents.

For the transformation lead: the barrier to scale is not model strength. It is whether the acceptance criterion for a new use case can be written in one day. A use case whose criterion cannot be written should have its rules settled on paper before engineering touches it, or engineering will take the cheapest route and turn ambiguity into a pass.

For the AI application lead: the eval set should grow out of failures, not out of a textbook. Fifty real failure snapshots are worth more than two thousand synthetic question pairs. Synthetic data tests whether the model can talk. Failure snapshots test whether it can be wrong inside your business.

For the compliance lead: the judgement record is the audit trail. Compliance does not need more logs. It needs every release of an agent action to carry a signer, a criterion and a way to recompute. Logs prove the system ran. Judgement records prove it was right. Regulators treat those as different objects.

Our judgement

First, what blocks scale this year is the ability to define done, not model capability. This is falsifiable. If the real constraint were the model, switching off observability across a batch of use cases should visibly degrade delivery, and switching off evals should raise rework. We ran the first half internally. Delivery did not get worse when monitoring went dark. We only found problems two days later.

Second, the thirty-seven points are a structural feature of the industry, not a maturity difference. The gap will not close on its own as models improve, because the gap sits at the definition end and definition can only come from someone who knows the business. Any claim that a stronger model arrives with its own evals skips that step.

Third, the cost of judging can only be paid by the business side. Engineering cannot substitute. What engineering can do is make a criterion executable. What it cannot do is decide what right means. That division is how our own product is built: the client writes the criterion, we write the recomputer.

When not to build this

Not at low volume. At a few dozen calls a day, manual sampling is cheaper than an eval set, and the eval set would sit unused.

Not when the outcome is machine checkable. Reconciliation, compilation, staged rollout and risk interception already own their criteria. The compiler, the variance report and the rollout metric are the eval. Do not build a second one.

Not before the business agrees on the rules. If the rules are still contested, an eval set only freezes the disagreement into a score, and everyone argues about the score instead of fixing the rules.

Three steps for today

First, write one acceptance sentence per delivery class. One sentence, readable by someone else, checkable by a machine. If it cannot be written, that work should not be handed to an agent yet.

Second, turn done into something checkable: an artifact path, a reproduction command, or a hash. Keep one copy before the change and one after, so anyone can rerun it.

Third, take five snapshots from last week’s rework and start a regression set. Do not wait for the model upgrade day to start collecting cases. They will not be there.

One question for the buyer: of the agents your team shipped last quarter, which delivery can an outside party independently reproduce in ten minutes? If the answer is none, what you own is not production capacity. It is a set of healthy looking dashboards.

Sources: LangChain, State of Agent Engineering, June 2026, 1,340 responses collected 18 November to 2 December 2025. langchain.com/state-of-agent-engineering. Internal operating figures come from OOMeta task queue records.

FAQ

Why can observability be installed while evals cannot+

Sampling carries no business logic, so one collector serves every agent. Criteria live inside the workflow: a person must say what right means, and a vendor saying it for the client is defining the client’s business.

Where do the 89 per cent and 52 per cent figures come from+

LangChain State of Agent Engineering, June 2026, 1,340 responses collected between 18 November and 2 December 2025.

Can the 57 per cent in production figure be extrapolated to traditional enterprises+

No. The sample is 63 per cent technology, 49 per cent companies under one hundred people, and 9 per cent above ten thousand employees. It shows technology firms and small teams move fastest.

What does 95 per cent accuracy mean at production volume+

Ten thousand daily calls produce five hundred errors a day. At six minutes of human handling per error that is fifty hours, or six people employed by the error rate. Accuracy is a sampling metric.

How should an eval set be built+

From failures. Each snapshot records the input, the tool call sequence, the judgement and the reason a person called it wrong. Fifty real snapshots beat two thousand synthetic pairs.

Which use cases need no eval set+

Those whose outcome is machine checkable, such as reconciliation, compilation, staged rollout or risk interception. The criterion already exists and a second eval is duplicate work.

Run the same check on your own execution log

Send a redacted 30-day agent execution log and get an eight-row table in 48 hours: one row per technical item, each cell carrying its numerator, denominator and criterion, gaps sorted by whether a schema change is needed. First three engagements are ¥8,000.

See the field check →