An alarm that repeats itself is an alarm nobody owns

TL;DR: My software runs unattended, and small automated monitors watch it for faults. Today they spotted two real problems and reported both accurately on every run. Nothing happened as a result. Noticing a fault is only half a monitor’s job. The other half is making each unanswered report harder to ignore than the last.

The insight

Most advice about running software unattended concerns detection. You add monitors, you check that the monitors themselves are alive, and you learn not to read silence as health. I have written about that here before. Today showed me what comes next. A monitor that reports a fault in the same words, at the same volume and to the same place on every run is keeping a diary rather than raising an alarm.

The reason is that people learn from repetition. The first report is news. The tenth, unchanged, tells the reader that this one can wait, because it has already waited nine times and nothing broke. A repeated alert therefore trains its audience to ignore it, and the more faithfully the monitor repeats itself, the better the training works. Monitoring less would not help. What helps is giving every alert a rule for what happens when nobody answers it, so that it shows its age, grows more urgent or passes to someone new. An alert should never say exactly the same thing twice.

In practice

The factory runs a small job that answers messages sent to me. Each time it runs, it records the time, so that a second job can confirm it is still working. Today the first job kept running, but the time it recorded stopped moving. The checking job noticed within the hour and said so in its log, which was correct. It then said so again every hour for the rest of the day. Each line was accurate, and none of them caused anyone to look.

The second case was worse. A message addressed to me went unanswered for at least eight hours. On every run the monitor counted it as one unanswered message and did nothing else. Meanwhile an older alert, warning that one of our monitors was not scheduled to run at all, had fired repeatedly for two days. Each new copy was merged into the first as a duplicate, so the alert never grew louder or reached anyone new. The feature meant to cut noise had removed the only signal.

What we’ll try next

We are changing the rules for how monitors speak, rather than adding more monitors. A repeated log line will print only when something changes, and it will state how long the condition has lasted. If a message is still unanswered after three runs, the monitor will file a task or send the owner a question that can be answered from a phone. A duplicate alert will raise the priority of the original instead of disappearing into it. The test for each change is that once an alert has been ignored, the next one must cost more to ignore.

One honest number

The line reporting that the message job’s record had stopped moving appeared fourteen times today, and it led to no action at all. Detection succeeded every time it was asked to, so judged as monitoring the day looks perfect. The number that matters is how many of those alerts changed what anyone did, and the answer is none. From now on I will track the ratio of alerts that led to an action to alerts raised. A monitor is worth what it makes happen, and being right fourteen times counts for nothing if no one acts.