Silent by config
nine hundred runs that failed unseen

At seven in the morning on November 13 the system health self-check finished and came back red. The duplicate database was zero bytes, and the check layer had written a line into its own output saying this would be misread as the collector not running. That line sat in the output from seven in the morning until the afternoon. Nobody was woken up.

Three more things happened the same day. The pre-market data prefetch exited one at the stop-loss pricing step, so the risk positions were not refreshed. The late-session reminder lost its connection at two in the afternoon and a rerun five minutes later succeeded, so the first failure went unrecorded. The sales heartbeat repeated the same thirty-two lines of an unregistered outbound error every six hours for thirty hours. Four incidents, three kinds, all present in the incident table, and the alerted-at field empty on every one.

The numbers in thirty hours

The window runs from six in the evening on the twelfth to twenty-two minutes past six on the thirteenth. In those thirty hours nine hundred and thirty-one runs were marked silent, meaning no delivery, and seven of those outputs contained explicit ERROR or FAIL lines. Four incidents sat at detected but not alerted. A fifth, an authentication failure from October 12, is still marked detected with an empty alert time, thirty-three days on.

The comparison matters. The same class of failure on November 8 and 9 carried an alert time. The path broke later; it did not never exist. The same system could wake somebody a month ago and could not now, and what happened in between was a configuration migration rather than a code crash.

Silence on its own is not the problem. A frequent check with stable output and no state change should not make noise. The problem is the trigger, and the trigger stopped pointing at the thing it was supposed to watch.

The trigger watches the wrong object

Across the three steps, record, judge and alert, the third step tests whether this round should make noise rather than whether this round came back red. The two questions look close and point at different things: the first is a property of the job, the second is a result of the run. Once the property is written as silent, every red state that job will ever produce inherits the silence.

The incident table treats recorded as notified. The row is in the table, the alerted-at field is empty, the interface reads detected, and everyone assumes somebody knows. That misunderstanding needs nobody to make a mistake. It only needs nobody to query the empty field.

This is not a one-field bug. Every new silent check adds one more red state with no exit. The system grows, the path widens, and it widens on its own as jobs are added. Nobody has to do anything wrong for it to happen.

Why one more alert rule does not fix it

The usual repair is a rule on the alerting platform: notify when a job fails. That rule works for services that emit metrics and does not work for unattended batch jobs, because failure there is not a change in a metric, it is the absence of a series. Metric systems have a function for exactly this. Prometheus ships absent and absent_over_time, and the documentation says they are useful for alerting when no time series exist for a given metric name and label combination.

Absence becomes a first-class failure only if you first define what should have been there. That is where the problem actually sits. Observability conversations usually start with what to collect, while the first question for an unattended system is who defines the expectation and where it lives. An expectation held in somebody head is not an expectation, because nobody can take a difference against a memory.

One more layer is easy to miss: the absence window is itself a cost. A short window produces false alarms on normal variation; a long one stretches detection latency. Of the four incidents inside thirty hours, a six-hour window would have caught the shortest only six hours late, while the late-session reminder carries about ten minutes of business value.

The expectation has to become a queryable table

Three arrangements were tried. The first lets every job declare who to notify on failure, and the problem is that whoever adds a job forgets to declare it, and the default for forgotten is silent. The second puts rules in a central alerting platform, with the same flaw: rules work on metrics, not on absence. The third has every run write a row saying it ran and produced this result, plus a reconciliation job that compares against what should have run today. The difference is the failure.

What matters in the third is not the reconciliation. It is that the word should must be enumerable. Somebody has to write the plan as data: which job, at what cadence, by what time it should have produced output. There is no technical difficulty in that. The difficulty is that it asks operations to move its memory into a table, and that table needs an owner.

This round was fixed that way. When a silent job produces a red state it must break silence or become a task card in the queue. The acceptance test is that incidents sitting at detected with no alert drop to zero inside six hours, and stay there two days running. The test is written to be countable on purpose, so a script can check it instead of a person remembering.

The delivery channel breaks too

Breaking silence depends on one premise: the channel is alive. When the channel is down, the job goes red, the judgement is correct, the message cannot leave, and the result is identical to silence. That dependency stays invisible until the day both fail together.

So the fallback cannot be built on the channel. A reconciliation digest now goes out at a fixed time every day, including the days with no failure, one line stating how many runs were expected, how many happened, and that the difference is empty. Its value is not notification, it is proof that the path still works. A no-anomaly line that arrives on time every day is more reliable than an alert that only arrives when something breaks, because the second cannot prove itself.

Output structure decides whether anything can be scanned

This investigation ran into a second problem. The last thirty hours produced five hundred and one task outputs totalling one million one hundred and fifty-eight thousand characters, and six hundred and fifty-five thousand of those characters were the task prompt echoed back into the artifact, fifty-six percent of the total. Two consequences follow. Any scan looking for failure text hits a wall of prompt prose first, and the real error line sits underneath it. Knowledge archiving files the prompt as content and stores duplicates.

That is a structural defect rather than a wording problem. Outputs should separate prompt from result explicitly, otherwise the signal-to-noise ratio of every scanner is decided by how much the author happened to echo. The repair is plain: fix the prompt section as a quoted block in the template and scan only the result section, or stop keeping the prompt at all.

What the misses cost

A missed incident can be priced. Twenty to forty minutes of triage, plus the delay downstream. Among the four, the late-session reminder would have left an entire batch of position alerts unsent if nobody had rerun it, and the health self-check failing two days in a row means the collection path was broken for two days before anyone knew, with a data gap that then needs other evidence to fill.

Spread the four incidents across a month at thirty minutes and two hundred yuan an hour and the direct cost is four hundred yuan. That number is small enough to ignore, which is why this class of problem never gets fixed for cost reasons. The real cost sits elsewhere: an authentication failure that has sat untouched for thirty-three days means the incident table has lost its meaning, because nobody reads what it records.

Limits

The approach has a scale premise. Nine hundred runs in thirty hours is a mid-sized company. At hundreds of thousands of runs a day the write amplification and the set difference need their own design, usually sharded by job and cleared on a rolling window. The mechanism stays; the engineering changes.

The other premise is a stable cadence. If a fleet changes schedule daily, what should have run has to be recomputed every time, and reconciliation costs catch up with its benefit. Those teams should freeze cadence first and reconcile after.

One class of system does not fit at all: online services that must answer within seconds. An absence window measured in minutes cannot save a three-second timeout. Those need a different layer, continuous sampling of latency distribution and error rate.

Our judgment

In an unattended system the smallest visible unit of failure is not a line in a log. It is the difference between expected and actual. A log needs a person to read it; the difference can be computed by a job. That single property decides whether a system can be discovered while nobody is on night shift.

Consequence one: if a team alerts only from job output, every new job adds one more silent risk, because notification is an optional property of the job. Consequence two: if alerts derive from a plan table, a new job adds one line of expectation, and the failure mode becomes forgetting to add it, which the reconciliation itself can catch.

Consequence three is counter-intuitive. Silence is correct design; a routine check should not make noise. The problem is not silence, it is that silence swallows a red state. Two unrelated decisions are controlled by one switch, which is a design error rather than an on-call error. Split the switch in two, one for whether to speak during normal operation and one for whether to speak when red, and the mechanism is fixed.

Three actions

First, count the rows in the incident table where alerted-at is empty and state is still detected, and look how long the oldest has been sitting. That count is your silent inventory, and it is usually larger than estimated.

Second, write the plan as data: job, cadence, latest expected output time. Even a CSV gives the reconciliation something to compare against, and the difference something to mean.

Third, separate prompt from result in the output template. It has nothing to do with alerting, and it decides whether the next investigation can find the real error line.

A question for whoever is on call: when was the last time a failure notification woke you. If the answer does not come, the system may be red right now and nobody knows.

References

Internal figures come from our own task outputs and incident table over the window from 18:00 on November 12 to 18:22 on November 13, 2026. The external mechanism reference is the Prometheus query function documentation.

Prometheus query functions: absent and absent_over_time, useful for alerting when no time series exist for a given metric name and label combination