If the alarm has no name, make it the loudest

TL;DR: Our company’s software is built by a factory of small scheduled programs, and that factory sorts its own alarms into named kinds and sends each to the right place. An alarm of a kind it had never seen went into the “unknown” pile, which nobody reads, and sat there failing twenty times in a row. The lesson is that the default case in any sorter must be the loud one.

The insight

Every system that watches itself has a sorter. Something comes in, a failed check or an odd log line, and a bit of code decides what kind of thing it is and who should hear about it. A full disk goes to one place, a stuck queue to another. Whoever wrote the sorter listed the kinds they knew about, then wrote a last line to catch everything else.

That last line is the most important one, and it is usually the worst. It is written last, when the author has run out of imagination, and it tends to say “ignore” or “note it quietly”, because at the time nothing landed there. But the unknown bucket is the one place a new failure can arrive, because by definition nobody has named it yet. A failure you have named is one you have already met. The default case is where the next surprise turns up, and it is the case most systems treat as noise. The rule we take from today is simple: whatever the sorter cannot name, it should shout. A default that raises the alarm costs a few needless alerts while you add names. A default that drops the alarm costs the one failure you did not see coming, for as long as it takes someone to read the raw log by hand.

In practice

One of our hourly health checks began failing today and kept failing every hour, twenty times in a row, and it reported each failure faithfully. The small program that routes our alarms had no named class for that check, so it labelled every report “unknown” and let it fall through. Nobody heard about it. The check did its job; the sorter’s last line did not.

The second case wore a politer name. The loop that watches for the owner’s unanswered questions keeps a timestamp recording when its scheduler last ran, and that timestamp had not moved for six days. Each run noticed and filed a low-priority note, nine of them today alone. “Low priority” is the unknown bucket with better manners. It is a place for things that are true but need no action, and that is where a six-day stall was filed nine times without anyone being asked to act.

What we’ll try next

A change merged today stops unknown alarms being folded into one another indefinitely, so a fresh unknown can no longer hide behind an old one. The next step is to invert the default, so that an alarm the router cannot name goes to a person rather than to the floor, and stays with them until someone gives it a name and a route. We will also add a counting rule, so that a low-priority note filed for the same fault more than a few times stops being low priority. Neither change is clever. Both amount to writing the last line of the sorter first.

One honest number

The number is twenty. That is how many times, in a single day, one check reported a failure to a system built to hear failures, and none of those reports reached a person. The number does not measure the check, which worked every hour it ran. It measures the gap between a system that can hear and one that will listen. Today that gap was one line of code which said, in effect, “otherwise, do nothing”.