A 9,136-run alert that never named an item

An automated check scans our backlog every fifteen minutes and pulls out the items that are past their deadline. It has run 9,136 times since June. Every run produces the same sentence with different numbers: which files it looked at, and how many items are overdue. Not once has it said which fourteen items those are.

So the alert kept firing and kept changing nothing. Anyone reading “14 overdue” could not tell which item to pick up, who owned it, or how long it had been sitting there. It refreshed every fifteen minutes for a whole day and not one item moved. Detection was never the problem. The output path was: the itemized list exists in the code, hidden behind a verbose flag that the scheduler has never passed.

Nine thousand runs, not one name in the log

The mechanism is plain. The check first computes the set of items whose deadline has passed, then decides what to print based on a flag. With the flag off, it prints the size of the set. With the flag on, it prints the contents. The default is off, and the caller is a scheduled job whose arguments do not include it. So on every run the detail is computed in memory, then discarded before the log line is written, leaving a single number behind.

This kind of defect is hard to notice because it looks healthy on every conventional signal. The job runs on time. The exit code is zero. The output is not empty — there is a fourteen in it. The frequency is steady. If you only watch for whether an alert fired, and whether it fired on time, you will never see anything wrong. What was discarded is not inside the number. It is behind it.

We turned the flag on and confirmed it by hand. On a single run, all fourteen overdue items came back with names. Some of them explained their own urgency on sight: a risk-limit refresh failing silently, where one bad instrument could stop the refresh across the whole book; and a price on an execution card that was structurally invisible to automated monitoring, which had already missed six execution windows in a row. The other nine were the usual families — self-attestation gaps, fields that contradicted the facts, context truncated without a warning — and all of them had already been diagnosed once.

Why a count is not an alert

For an alert to lead to work, the reader needs three things: the name of the item, the owner, and a decidable next step. Remove the name and the response degrades into “go through the backlog.” The cost of going through the backlog is exactly the cost the alert was supposed to save. It has not disappeared; it has moved from a machine to a person, and it moves every fifteen minutes.

A count has a second property that is harder to see. A constant number reads as a steady state. Fourteen shows up every day, and readers quickly file it next to the hum of the server room. When it becomes twenty-five, it is a louder hum, not an event.

Nobody here was slacking. The check did precisely what it was asked to do: report a count. The output contract said count, and it delivered a count, flawlessly.

That also explains why “add another check” so often fails as a fix. A check is code, and its ceiling is set by its output contract, not by its detection rate. A check with perfect detection and unactionable output is indistinguishable in effect from a check that does not exist, at a higher price.

The second kind of self-muting: warn only, never block

The same system carried a second defect of the same shape, pointing the other way. The capacity warning on the queue came with a comment in the code: warn only, never block. When the day’s intake crossed a threshold, the system raised a notice and refused nothing. The notice was recorded, the queue kept growing, and within three months the number of notices and the length of the queue had become the same background hum.

“Warn only” and “count only” look like different problems, one dropping the names and one dropping the constraint. The mechanism is identical: a decidable verdict gets downgraded into information, and the information is handed to a place that will not act on it automatically. Once a verdict becomes information, it may as well not exist.

That is why we changed both in the same round. The alert and the capacity constraint are two cross-sections of one chain; fixing one leaves the other to regenerate the problem.

One boundary is worth drawing. Not every alert needs a name. A scalar threshold — disk filling up, certificate about to expire — is actionable on its own, because it points at one thing rather than a set. Names become necessary when the alert’s object is a set: each item costs differently, belongs to someone different, and needs a different next step. Compress that set into one number and the most expensive information is exactly what you lose.

Nine of the fourteen had been closed three times before

With the detail on, the second finding was worse than the first. Most of these overdue items were not new discoveries. They had been diagnosed, and ticketed, and then allowed to age into silence. Over three months, the same class of problem — a backlog that reports without digesting — was dealt with and closed at least three times: twice in June, once in July, twice in September.

Every closure looked the same: a person went through the queue, removed the duplicates and the temporary items, and closed it. The mechanism never changed. So the same path reappeared after each closure. The latest reading: thirty-nine items opened in seven days, sixteen closed, a net gain of twenty-three.

The cost of this loop is easy to total. The check runs every fifteen minutes: ninety-six times a day, 9,136 times in three months. Assume a one-in-ten chance that any given run makes a person stop and look. That is over nine hundred interruptions of attention, against zero items moved in those three months. This is not a system running below its potential. It is a chain with a negative net value, running reliably.

There is one more layer worth stating separately. The part of the system that finds problems and the part that digests them share a single queue. The fuller the queue, the longer a fresh conclusion waits inside it, and the more likely it is to expire while waiting. More output makes the entrance tighter; a tighter entrance produces more expired items. Several of the fourteen above were opened by the side whose job is diagnosis. Nobody was failing to try. The loop is built out of a shared entrance.

Put the two findings together and the hard part of the fix is not technical. Adding a flag is one line of code. What is hard is admitting that 9,136 runs produced no conclusions because we had been treating reporting as delivery.

What we changed

First: the alert carries names, but without becoming noise. Fourteen lines every fifteen minutes is not a fix, just a different kind of noise. So detail is emitted when the set changes, or as the first few items plus a total, with the full list written to the day’s archive where it can be pulled up.

There is a design question to answer before any of the noise control works: where the granularity of change sits. Triggering on confirmation state makes the alert flap whenever one item is re-confirmed. Triggering only on set membership goes too quiet and can miss a qualitative shift, such as one item crossing thirty days overdue. Which one you want depends on which kind of change costs more in your queue, not on taste.

Second: an overdue item must have an exit. There are exactly three acceptable dispositions — do it, downgrade it, or explicitly discard it. A queue with no exits does not converge; it accumulates. That is the direct cause of the recurring path above.

Third, and the one that matters: the output contract moved from reporting a count to delivering work that can be picked up. The first two changes are how that contract is implemented. One lesson about defaults while we were here — the flag had been off for an entire quarter, which means the default was the product decision all along. Changing a default is worth more than adding a switch.

We should also admit what is unfinished. Overdue items now have names and exits, but capacity is untouched: diagnostic output still enters the same queue. That is the next layer, it costs more, and we do not have an answer yet.

Three checks you can reuse

If your agent operations also run automated checks, three questions go a long way.

• Open the most recent alert output. Can it answer which item, whose, and how overdue? If not, it is noise, no matter how punctual it is or how many times it has run.
• Open the queue. Do overdue items have defined exits? A queue with reminders but no three-way disposition will accumulate, at a rate proportional to how much work you produce.
• Look at the defaults. Detail off by default, blocking not enforced by default, export not moving by default — defaults are the real product behaviour. Documented capability is not.

An alert is worth what it converts into named, owned work. The next time you see one that reports a number and nothing else, ask a single question: if it went quiet tonight, would anyone notice? If the answer is no, it was not working now either.

References

• Google SRE Book — Monitoring Distributed Systems
• Run counts, overdue-item counts and queue readings come from our own operational records and can be recomputed field by field.