A task should be the size of its fix

TL;DR: We run a software factory whose tasks are filed by the programs that watch it, and a monitor files what it sees, which is a symptom. Today one fix closed about seventy tasks. That is not seventy pieces of work, and a queue that says it is will mislead everyone who reads it.

The insight

We run a software factory: a business whose daily work is done by programs, with a person setting direction and nobody doing the tasks by hand. In such a place the people do not file the work. The monitors do. A monitor watches a log or a queue and writes a task when it sees something wrong. This is the right design. Nobody is awake to notice, so the noticing has to be a program. But a monitor sees symptoms. If every symptom becomes a task, the queue stops describing the work and starts describing the monitor’s eyesight. Ten symptoms of one cause become ten tasks, and the queue is ten times longer than the work.

The fix is a rule about size. A task should be the size of its fix, not the size of what a monitor saw. When a monitor sees a new symptom, the question is not whether the symptom is new but whether an open task already covers the change that would make it go away. If one does, the symptom attaches to that task as evidence. If none does, a task is filed for the cause and the symptom becomes its first exhibit. Done this way, the length of the queue means the number of changes still needed. The few faults that do need a change of their own are no longer buried under copies of one that does not.

In practice

We run a loop that watches for recurring failures and files a task for each distinct fingerprint, by which we mean each error message whose wording differs from the others. Today one change taught the loop to classify those fingerprints properly, and it closed about seventy tasks at once. For days before that change, the queue had carried seventy items for one piece of work, and our retrospective had counted them as seventy closes.

A smaller example hid work rather than inflating it. A housekeeping step in the task store failed three times in a row, raised an alarm, then succeeded and withdrew it. Within a couple of hours the same failure returned, and the same alarm was raised and withdrawn again. Each episode got a task. The cause, which had not changed between them, got none. Filing by what was seen gave us two closed tasks and one open fault.

What we’ll try next

We have filed two changes and will make a third. The loop that files recurrence alarms will group new fingerprints under one parent task per suspected cause, so that the classification work reads as one task from the start. An alarm task will stay open until the failure has been absent for a stated period, rather than closing the moment the symptom clears, so that a cause which keeps returning shows up as a task that stays open. We will also make the relay that turns chat messages into tasks check what it has already filed in the same thread before it files again, because it produced duplicates today for the same reason.

One honest number

The factory closed 371 tasks today. About 70 of them were the single-fingerprint alarms that one change cleared, so roughly one close in five was the same fix counted again. A reader of our daily count would have put the day’s work a fifth higher than it was. A reader of the queue the week before would have thought the factory seventy items behind when it was one.