After a 12-day outage
three losses nobody noticed

The machine had been down for 12 days. When it came back, the scheduler spread every missed job into one burst. Inside five minutes, six of them were rejected by the model provider for making too many requests. Four re-ran later. Two never did, and that day produced nothing at all. No alert fired.

A crash shouts. A stop does not.

The two jobs that produced nothing were the daily 19:00 strategy radar and the content engine that runs every eight hours. We found out the next morning, when someone asked in the channel why there had been no report.

Both failures landed exactly in the blind spot of our monitoring. Alerting rules are built on two signals: is the process alive, and did it exit with zero. A silently stopped job is normal on both. It simply has no result. The monitor was watching whether the job ran, not whether it produced anything. Those two questions agree almost all of the time. When they disagree, the screen is calm and the output is zero.

Our judgment: in a system that runs itself, the most expensive failure is not a crash. It is a stop that reports nothing. A crash leaves an exit code, a stack trace and an alert. A stop leaves an empty slot in the calendar. The only hard evidence that a job is really running is whether today’s output exists.

Loss two: the precondition vanished, the script exited quietly

A job that refreshes a monitored account list had failed every Monday morning for three weeks. It needs a browser that is both logged in and exposing a debugging port. No such browser exists on that machine any more. The script cannot connect, exits, and writes nothing.

Three weeks later we opened the list. It was still the September 21 version: 45 accounts. Nobody had mentioned it, in any conversation or any report. The real defect was not in the code. It was in the environment: for the script, “no browser to talk to” means “nothing to do today”.

Jobs like this stay green in every health check, because the failure looks like success. The fix is specific. Write the external dependency as an explicit check, whether the login state is present, the port is open, the quota is left, and when the check fails, leave a record before exiting. One check and one log line cost far less than three weeks of silence.

Loss three: stale treated as resolved

The same day, eight medium-priority tickets were auto-cancelled in the same second for sitting too long. On the evidence, at least two underlying problems were unfixed. One was an instruction file for an agent that had grown past the context limit and was silently truncated, so the job finished while reading half the rules. The other was a symbol list for minute-level quotes that excluded the watchlist, which locked the coverage of a stop-loss engine’s similar-day simulation. Its downstream ticket is still open.

The cancellation record says auto-cancelled. Read quickly, it looks like something was handled.

Our judgment: an expiry rule solves a queue that is too long, and it uses age as a stand-in variable. A timestamp proves nobody touched the ticket. It does not prove the problem is gone. Treating the two as the same thing means letting the system announce a conclusion it has no evidence for. Automatic closure should separate handled from unseen: the first needs evidence such as zero remaining downstream references or a passing check, the second can only be marked for review.

Three losses, one gap

All three share one missing record. Jobs keep an account of whether they ran, and no account of where the output went. Put the test on the process and losses at the level of results stay invisible forever.

Move the test from the process to the output

None of this needs a new system. Five steps in order are enough.

External evidence points the same way. In the IBM Institute for Business Value survey of 2,000 technology executives published on June 8, 2026, 70% said business teams deploy technology faster than IT can track it. Salesforce’s May 2026 survey of 2,025 decision makers found that most deployers went live with only part of their oversight in place and built the rest afterwards. Both are saying the same thing: more things run than have an owner. An output ledger is the cheapest way to bring those two counts back together.

Where this stops working

First boundary: some jobs leave no file, a message sent to a channel, for instance. Even then, the fact that it was sent has to be recorded, or it is indistinguishable from never having run.

Second boundary: an output ledger proves a job ran, not that it ran correctly. Correctness needs eval sets, sampling and a release gate. Mixing the two gives you a busy-looking ledger.

Third boundary: the check itself breaks. Ours writes one line a day saying how many jobs it inspected and how many were missing output. On the day it dies, that line disappears, and the absence is the signal.

The consequences differ by role. An AI application lead gets a check script that can be written today. A transformation lead gets a new acceptance test: today’s output, not the number of platforms provisioned. A C-suite sponsor gets the number that should appear in the daily report: outputs, not jobs. A compliance lead gets a gap record, because days with no output need a written reason or they cannot be explained afterwards.

Our judgment

Making a stop audible costs an order of magnitude less than making the system never fail. We wrote the rule into our own operations: a job declares its output before it starts, and the output is checked on disk before the job counts as done. What changed was not the number of failures. What changed is that failures no longer need to be discovered by accident.

Tomorrow morning, open your scheduled jobs and ask each one where yesterday’s output is. The ones with no answer are the ones that will disappear the next time the machine goes down.

References: the three operations facts in the first half come from OOMeta’s own run records. IBM Institute for Business Value, CIOs and CTOs Face Growing AI Control Gap, June 8, 2026: newsroom.ibm.com/2026-06-08-new-ibm-study-finds-cios-and-ctos-face-growing-ai-control-gap-as-enterprise-deployment-scales Salesforce, State of Agentic AI in the Enterprise, survey of 2,025 decision makers, May 2026: www.salesforce.com/news/stories/agentic-ai-leaders-survey-on-roi