Why AI agents fail in production: a log audit

At 20:01 on 23 September 2026, one scheduled agent in our fleet had a tool call return a Python traceback. Scrolling up the same file, 88 command timeouts and 127 provider quota-exceeded events were already sitting there. Three failures, one file, three entirely different fixes.

What we counted

This is the runtime error log of OOMeta’s own agent fleet, covering 2026-09-21 07:18 to 2026-09-23 20:01 CST: 2.6 days, 16,787 lines, 139 scheduled sessions, 1.4 MB. The log rotates by size, so this is a window rather than a full history. Classification is plain string matching, so anyone can re-run it.

Why be this precise about the caliber? Because a single number labelled agent failure rate can differ tenfold depending on the caliber behind it, and most dashboards publish the number without the caliber.

Failures sit in five layers, not one plane

Tool layer: 423 tool calls returned an error. By tool, terminal commands account for 267, or 63 percent; code execution 46, file search 36, skill maintenance 29, file reads 15, patches 10.

Transport layer: 222 lines of SSL or connection resets, 80 lines of HTTP 429. Its signature is randomness: the same request usually succeeds when resent.

Quota layer: 127 weekly-quota-exceeded events across 33 sessions. One in four of the 139 sessions hit a quota wall. This layer never fails a connectivity probe; it only blocks full-sized requests.

Script layer: 385 lines of Python traceback. The quietest layer, because it does not stop the session. It just fails the same way every time until someone reads the log.

Policy-gate layer: 13 commands blocked by a safety gate. That is a policy working, not a failure, but it lands in the same file and inflates the denominator by default.

Model layer: 0 entries.

The zero is the finding

Not one entry comes from model output quality. That cannot be read as the model never erring. It reads as that layer not being inside the observation surface. On the same data, a team watching only the model sees 0 and a team watching only tools sees 423. Both numbers are true and neither is the whole picture.

Which is why agent instability as a finding carries almost no information. It averages five layers whose owners, fixes and budgets all differ: transport wants retries, scripts want code changes, quota wants a procurement conversation, and the policy gate should never have been in the denominator at all.

Our judgment

1. Layered counters. Keep one count each for tool, transport, quota, script and gate. Collapsing them into a single failure rate is like keeping the receipt and throwing away the repair order.
2. Model quality belongs to an evaluation set, not the error log. The log owns runtime faults, the eval set owns output correctness. Mixed together, the operations team tunes models while the model team reads timeouts.
3. Every counter must answer one question: is this layer healthy right now? If it cannot, it is decoration, and decoration never wakes anyone at 03:00.

One question for buyers

What is the denominator behind your agent failure rate? If it is all requests, then the fact that 63 percent of failures come from a single tool gets averaged away. If it is successful tasks, at least you know what you paid for one finished result.

Caliber and sources: every number here comes from OOMeta’s own agent fleet runtime log, first-hand, with no third-party vendor data. To reproduce: the log file at ~/.hermes/logs/errors.log, retained window 2026-09-21 07:18 to 2026-09-23 20:01 CST (1.4 MB, 16,787 lines); classification uses fixed string matches for exceeded the weekly usage quota, Command timed out, Tool returned error, Traceback and Blocked; sessions are de-duplicated on the session identifier in each line; collected 2026-09-23 20:10 CST. The window shifts as the log rotates, so a fresh run will not reproduce these exact counts, only this caliber. External caliber references: the exit code 124 timeout semantics are documented in the GNU coreutils timeout manual at gnu.org/software/coreutils/manual/html_node/timeout-invocation.html (curl HTTP 200), and HTTP 429 is defined in rfc-editor.org/rfc/rfc6585 section 4 (curl HTTP 200). Both are terminology references; no figure here depends on third-party data.