Agent verification must sit outside the run

At two in the morning two processes wake up together. One writes the audit rows for the past hour. The other reads those rows and works out whether anything was odd. The writer had actually stopped three days earlier. The reader ran on schedule, finished cleanly, and reported normal. Nobody did anything wrong and nobody lied. What happened is that the two jobs lived in the same body, so when the body stopped, both stopped, and nothing was left to say that it had.

A shutdown with no witness

This happened in our own governance database. The last audit row sits at 2026-10-20T01:00:34Z and the next one appears at 2026-10-23T02:37:52Z, a gap of 73.6 hours with zero rows. Counted by day, 20 October wrote 94 rows and 23 October wrote 687. In the same window the scheduled jobs carried a status of error, and the daily report said zero calls in the last 24 hours.

The more interesting part is what happened to the alerts. Three freshness assertions failed on 21, 22 and 23 October in a row, and by design the system opened tickets automatically. Nobody consumed those tickets. They stayed open until the execution environment came back on 23 October, when the same recovered process read them and closed them by itself.

Put the roles side by side and the shape is obvious: production, recording, alert reading and ticket handling were four seats in one domain. They stopped together and woke together. During those 73.6 hours the system produced nothing, and it also never produced a single sentence saying that it was producing nothing.

The duration needs a label. It is a single-host self-test with one subject, not a customer case, and not something to quote as an industry number. Its value is not the 73.6. Its value is the structure it exposes: nothing can witness itself.

A perfect score is not a correct run

This is not a quirk of our own box. Splunk’s October roundup on agent observability cites an analysis of 58 τ²-bench traces that earned perfect task rewards. In 83% of them, 48 traces out of 58, at least one procedural requirement was violated. The reward function only looks at the outcome, and procedural violations do not enter the reward, so “it worked” and “it was done the allowed way” became two separate facts.

The same article carries harder evidence. A preprint on silent tool failures, arXiv:2607.19449, found that a tool call can return HTTP 200 with a payload that is empty, null or malformed. Across 396 valid agent responses, fabricating a result was the most common behaviour at 56.6%.

Join those three sentences and you have the shape of most production incidents: the monitor sees 200 and reports green, the model receives an empty payload, and everything downstream keeps moving on invented data. The failure happens inside the word success. That is not a blind spot in monitoring, it is a definition problem in monitoring. What you are watching is the success flag that the executor wrote for itself.

Sampled evaluation cannot stand in for evidence

Kognitos put this more directly than most vendors in a piece published on 20 October. A trace records what the model produced, not which approved rule permitted it. Evaluation is typically sampled. Retention windows are short. Those three together downgrade “we can investigate” into “we can describe”.

An auditor always asks about one transaction: why was this one allowed, which version of the rule was in force, who approved it and when. Sampling answers a different question, whether this class looks acceptable on average. There is no conversion rate between the two.

Their replay standard is worth copying verbatim: the same inputs under the same rule version must produce the same outcome. If a system returns different outputs for identical inputs, there is nothing to reproduce, and the record of one run carries no information about the next.

One more piece of corroboration, this time from a vendor’s own wording. In its 15 October press release, ServiceNow described audit trails as after the fact and placed the runtime authorisation decision somewhere else. This is not a judgement of their product choice. It is a confirmation of a positional fact: a record written after the action, however complete, does not substitute for the decision made while the action was happening.

We broke this line ourselves once

The same class of error showed up on a different pipeline of ours. A website release committed locally, pushed successfully, and the report said shipped. The site had not changed and readers were seeing a page from two days earlier. The problem was not git. The problem was that the sentence deciding whether the release had landed came from the party being checked.

We replaced it with external readings, three steps, none of which passes through the executor. First, read the repository ahead and behind counts, which must both be zero. Second, read the status code of each live URL. Third, read the rendered HTML and grep for a fragment of the title added in this release. All three run in a different process and read a different system’s output.

The third step is not optional. A 200 is not enough on its own, because a cache will happily serve a 200 for a page from two days ago. The server is telling the truth about the request and you are getting stale content. Freshness can only be established by a trace of the new content, never by a response code.

A clock dressed up as a counter

That same outage left a detail worth more than the duration. One dashboard column, unaccounted calls, read 1 the whole time. The column next to it, lag, went 23 hours, then 47, then 71.

It looked like the system was tracking something. In fact that number was the time difference itself, converted to hours and displayed again. It can only grow, and it can never fire an alert, because its definition has no threshold in it. A dial that only turns one way does not ring.

The definition flaw has been removed from the code path and the ticket is closed. What stays is the rule: a monotonic number is not a metric, it is a clock. On any dashboard, take the cells that can only go up and ask whether they count events or count time. Anything counting events returns to zero. Anything counting time never does.

Where the verification point belongs

Three things decide it. The first is position. The verifying process must not share a lifecycle, a host, a credential or a queue with the executing process. The cheapest version is a separate read-only consumer that eats only the external artifacts the executor leaves behind, runs on its own schedule, and keeps running when the executor is killed. The test is blunt: at the moment the executor is killed, does anything on the verification side disappear with it? If yes, they were never separate.

The second is the object. Accept external artifacts only. Another system’s status code, a row written under another clock, a file hash on disk, an external counter, a timestamp from a third-party probe. Fields the executor fills in for itself, success, done, completed, get treated as claims rather than evidence, no matter how consistently they are written.

The third is scale. Verification has to hold for every transaction, not for a sample, and the retention window has to be at least as long as the retention requirement on the business record. Where those two windows differ, the difference is the period in which you have a control you can describe but cannot evidence. That gap is invisible in normal operations and comes due in one payment the day it matters.

There is an organisational version of the same rule. The party that lets an action through, the party that can stop it, and the party that writes the account should not be the same person and should not be the same process. When all three collapse into one, the accounting is always what gets sacrificed, because it produces nothing at the moment it is done.

What this does not prove

The 48-of-58 ratio comes from a third party’s analysis of public benchmark traces. It is not our measurement. The 56.6% is a preprint’s self-report with no peer review. The 73.6 hours is a single-host self-test with one subject, not a customer case.

Kognitos is selling authorisation and replay capability, so its judgement favours itself and should be read with that in front. Splunk’s article is a vendor roundup in the same way.

What transfers is the mechanism, not the numbers. There are two: the recorder has to be far enough from the executor to survive it, and the object of verification has to be an external artifact. Neither depends on any vendor’s product or on any model version.

The other direction deserves honesty as well. For actions with no side effect, that roll back cheaply and cost almost nothing per run, a shared domain is a reasonable trade, because the cost of being wrong is smaller than the cost of building a thick verification layer. Payments, external communications and writes into production systems are the three categories where the trade should never be made, because their errors do not reverse.

Our judgement

The reliability ceiling of an agent system is set not by how strong the model is but by how far the verification point sits from the execution path. Where that distance is zero, incidents do not arrive looking like incidents. They arrive looking like normal operation, and they continue until somebody outside the system runs into them.

Three things follow, all of them doable this week. First, draw the four-role map: production, recording, alerting, handling. Mark every resource any two of them share, one red line per shared resource. A system with more than two red lines has no verification layer, it has a hope.

Second, for every key metric add one sentence: when this breaks, who finds out, and how long does it take. Any metric that cannot answer that sentence should be deleted. It occupies a dashboard slot and its only future use is being screenshotted as evidence after the fact.

Third, before go-live, run a suicide test. Stop the recorder on purpose and time how long until a person notices. That interval is your real observability radius. Most teams measure it in hours the first time while believing they operate in minutes.

The question to leave with buyers: if you turned off whatever writes your logs right now, who would call you within the hour? If you can name them, you have a verification layer. If you cannot, what you are holding is not evidence. It is luck that has not run out yet.

References

Splunk, “6 Best AI Agent Observability Tools for Production in 2026”, Jackson Wells, October 2026 (source for the 48-of-58 procedural violations and for failures hiding behind HTTP 200)

arXiv:2607.19449, preprint on silent tool failures (tool calls returning 200 with empty or malformed payloads; fabricated results in 56.6% of 396 valid responses; not peer reviewed)

Kognitos, “Agent Observability: A Trace Shows the Path. It Does Not Show the Authority.”, 20 October 2026 (traces record production not permission, sampled evaluation, short retention, replay standard; vendor article, author sells the capability)

ServiceNow press release, 15 October 2026 (describes audit trails as after the fact and lists runtime authorisation separately)

The 73.6-hour gap, the 94 rows on 20 October and 687 rows on 23 October, the three consecutive failing freshness assertions and the unaccounted-call counter stuck at 1 all come from our own governance database and daily reports. Internal self-test data, not to be quoted as an industry figure.