When the Monitor Breaks Before the Agent

Once agents run in production, the expensive wrong answer is rarely that the agent failed. It is that the agent did nothing, a confident conclusion produced by the monitoring layer itself. We hit two forms of it.

Failure one: the probe was the wrong size

Our provider balance monitor ran a 1-token health probe, and it stayed green. In the same window the error log recorded 18 weekly quota exceeded events inside six hours.

The fault was the probe, not the provider. Plans meter usage, and a 1-token request consumes almost none of it, so the check passes by design. Swapping it for a request the size of a real agent call, about 9,000 input tokens, returned HTTP 429 on the same endpoint immediately.

The rule is blunt: a health check has to be sized like production traffic. A probe three orders of magnitude smaller than real load tests connectivity, not availability.

Failure two: missing data read as a negative signal

The second failure is quieter. Our channel cadence monitor used the local publishing record as its only source and kept emitting the same verdict: one social channel silent for 18 days. That premise was escalated six times.

The platform-side page showed a post from that same day. The gap was not in the channel but in the instrument. The script that backfills manual posts into the local record lacked a dependency in the scheduler’s interpreter environment, so it exited non-zero on every run and the failure left a single warning line in a log. The record froze at September 5, and nothing in it could separate a silent channel from a broken backfill.

The fix has three layers. Pin the interpreter so the backfill no longer depends on how it is invoked. Persist a platform-side reading, and on read failure write an error rather than a zero or an empty value. Change the gate so the most recent publish is the maximum of the local record and the platform reading, and require any dormant verdict to be backed by a fresh platform reading for that same channel, otherwise downgrade it to unverified.

The failure path was removed by making it impossible for a dead instrument to produce a certain conclusion, not by adding a reminder to be careful.

The same failure class in other people’s data

This is not specific to us. Gravitee surveyed 919 executives and practitioners: 88% of organisations reported AI agent security incidents in the past year, while confirmed incident rates fell even as agent fleets doubled. The report attributes that gap to underreporting and detection failure rather than improvement. Observability analyses in 2026 estimate fewer than one in ten enterprise applications are fully observable. MLflow’s production monitoring guide names trusting agent self-reports as the most dangerous blind spot, because agents that hit errors tend to produce plausible output rather than explicit failure messages.

Put together, the bottleneck map for agent production is clear. Model capability, orchestration frameworks and unit cost are all moving fast. The layer that answers how you know it is right right now has barely moved.

Our judgement: three rules we would enforce

Three rules you can apply this week:

1. A negative conclusion needs a primary reading of the same object. Local records, cached values and the last successful result do not qualify. Absence of data may only produce unverified, never did not happen.
2. Size probes like production. Token count, concurrency and payload shape all have to line up, otherwise the green light only proves the network is up.
3. Make monitor failure alert on itself. If a dead collector and a quiet world look identical on the dashboard, the green light carries zero information.

A question for buyers

Can your agent dashboard tell no traffic today apart from the collector died today? If it cannot, the green light is not good news. It only means no bad news got out.

References: both incidents are first-hand records from our own agent operations (quota probe: 18 quota-exceeded events in six hours in the error log, a 1-token probe returning healthy while a request of about 9,000 input tokens returned HTTP 429; channel cadence: the local record froze at 2026-09-05 and kept emitting a dormant verdict while the platform-side reading showed a post that same day, caused by a missing dependency in the backfill script’s interpreter environment). Internal evidence: operations/scripts/provider_balance_monitor.sh and sales/logs/publish-reconcile.log | Gravitee, State of AI Agent Security 2026 (919 practitioners; 88% of organisations reported AI agent security incidents, confirmed incident rates falling as fleets doubled, attributed to underreporting and detection failure), gravitee.io/blog/state-of-ai-agent-security-2026-report-when-adoption-outpaces-control (curl HTTP 200) | MLflow, Monitoring Agentic AI in Production: 2026 Guide (trusting agent self-reports named the most dangerous blind spot), mlflow.org/articles/monitoring-agentic-ai-in-production-2026-guide (curl HTTP 200) | Mastra AI agent observability guide (aggregates the Gravitee figures and the observability coverage estimate; secondary), mastra.ai/articles/ai-agent-observability (curl HTTP 200). The survey figures are vendor-commissioned; we cite them for direction only.