Merged is not running

TL;DR: We merged a fix yesterday and closed the task. The programs running on the build machine were still the old ones. A merge is a promise. The only finish line is the running system doing something new.

The insight

Our software factory is a set of automated programs that write, review and ship code with no person watching. Such a system produces a great deal of paperwork about its own progress. A pull request merges, a task closes, a job appears in the schedule. Each record says that something was done, and each is easy to mistake for the thing itself. None of them changes what the system does. A merge changes a repository; a closed task changes a list. The only event that matters to a user, or to the next program in the chain, is the moment the running software behaves differently. That moment must be checked on its own.

The temptation is strongest in an unattended factory, because the records are what the factory reads. If one program asks whether something is fixed and the answer comes from the task list, the list will say yes long before the fix is live. So our rule is plain. A change is done when the running system shows it. Every step that only records progress, whether a merge, a closed task or a scheduled job, must be checked against the thing it was meant to change, by something that does not take the record’s word for it. Addy Osmani makes a related point in The Code Nobody Reads: as people stop reading each change line by line, trust must move to checks independent of whoever made the change. The merge is the author’s report. The installed program is the independent check.

In practice

Yesterday one of the factory’s programs fixed a classifier that had been raising false alarms about blocked work. The fix passed through root cause, design, review, build, test and merge. Each step left its receipt, and the task closed. Then the hourly check that compares the programs installed on the build machine with the freshly built ones reported, run after run, that the installed ones were older. The fix had been merged, reviewed and closed. It may not have been running at all.

The second case has the same shape one level up. The owner approves work by merging a small pull request, and a job is meant to watch for those approvals. We found the job had never been scheduled, so we scheduled it, and it now runs every five minutes. But its last step, the merge itself, fails on every run and reports nothing. The log shows the job reading the owner’s reviews and never shows a result. We fixed the absence of the job and counted that as done, without checking whether the job does its job.

What we’ll try next

We will make two changes, both aimed at the gap between the record and the thing. First, the task that carries a fix will no longer close on merge. It will close when the installed program on the build machine contains the fix, and a check that reads the running version will be what says so. Second, the hourly comparison of installed and built programs will stop merely noting a mismatch. If the mismatch survives two runs, the check will either install the built programs or raise a single alarm, rather than logging the same sentence for the rest of the day.

One honest number

Five hourly checks in a row reported that the installed programs on the build machine were older than the ones just built. Every one was logged and none was acted on. Five out of five is the number to keep. It says the factory saw the gap each time and treated seeing it as enough. That is the mistake this post is about, written out by the machine itself.