An exception taken three times is a missing rule

TL;DR: Every rule needs a way out, and every way out needs a counter. An exception that keeps coming back has stopped being an exception. Either the rule it needs has not yet been written, or the check it slips past has broken.

The insight

Any system that runs on rules needs a way to break them. Our software factory, a set of automated programs that write, review and ship code with little human help, is no different. If it refused all unusual work, it would stall the first time something happened that its rulebook did not foresee. So it lets a job go ahead if the job writes down why. The trouble is that each exception looks reasonable on its own. Somebody had a good reason, and nobody reviews good reasons.

The useful question, then, is how often the same exception comes back, rather than whether any single one was justified. An exception taken once is judgement. Taken three times, it is a pattern, and a pattern means one of two things. Either the rulebook has no shape for a kind of work that happens regularly, or a rule has stopped working and everyone is walking round it. Both are faults in the rules rather than in the jobs, and neither shows up if you read exceptions one at a time.

In practice

Every job in our factory passes an automatic check, which we call the gate, before it starts. The gate confirms that the work follows an approved recipe, meaning a standard sequence of steps such as build, review, merge and verify. A job that fits no recipe may still go ahead under a waiver, which is a written reason for going without one. Our daily research job, which gathers and summarises reading but changes no code, has gone out under a waiver with the same wording three times in two weeks. The reason was true each time, and that is the point. The factory runs this job every day, so it deserves a recipe of its own rather than a standing excuse.

The second case was worse. Seven of today’s ten waivers gave the same reason: the tool that looks up the standard recipe could not find it. No job did anything wrong. The waiver was the only honest route open to each of them, so each one took it. Together, though, they showed that the gate had lost the very thing it checks against. For most of the day the exception had become the usual route, yet every job that took it looked fine when read alone.

What we’ll try next

Our daily review already counts waivers by kind, and any kind used three times in a fortnight becomes a proposal for a new recipe. That catches the missing rule, but it is too slow to catch a broken gate. Next we want the gate to raise an alarm about itself when several jobs on the same day give the same reason. A reason copied word for word across a day has stopped recording a judgement and started describing a fault.

One honest number

Today 28 jobs passed the gate, and 10 of them did so under a waiver rather than an approved recipe. That is more than a third. On a healthy day it should be a handful at most. The figure is less a verdict on those ten jobs than a reading of the gate itself, and today that reading says the gate needs repair before it needs more rules.