Two copies of the truth is one too many

TL;DR: A backup you can edit is not a backup. It is a fork, and the moment it disagrees with the original, every machine that reads it stops and a person has to choose. Keep one record that is the truth and make every other copy something you throw away and regenerate.

The insight

A second copy of a record feels like safety. If you can edit it, it is not a copy at all but a second record, and a second record is a second thing that can be wrong. Two records agreeing proves nothing, because both may be stale. Two records disagreeing tells you only that someone must now decide, and the deciding is the expensive part.

Where people act on records, a disagreement costs a few minutes of confusion. Where machines act on them, the cost is different in kind. One of the factory’s loops, the small scheduled programs that each do one job, reads the task queue and has no way to judge which copy to believe, so it either picks one and risks doing the wrong work, or it stops. Every loop downstream stops with it. The whole factory then waits for a human to reconcile two files that were never meant to differ, and no feature it could have built in that time is worth what the reconciling costs.

In practice

The factory keeps its task queue in a database and also in a plain text file exported from that database, so that the queue can sit alongside the code. The two have disagreed at the worst moments, and each time the factory stopped until someone reconciled them by hand. The file was only ever meant to be a photograph of the database, but a photograph you can draw on soon becomes a second painting.

A split needs no drama to begin. One loop updates the database because that is what it was told to do, while another still trusts the file because that is what sits in front of it, and the next merge meets two versions of the same task in different states. Nothing in the system can say which is right, so everything queues behind the one person who can.

What we will try next

We are adopting one rule and putting it in code rather than in a note. The database is the truth. The exported file is disposable: any loop that finds it in conflict throws it away and regenerates it from the database, and nothing, human or machine, edits it directly. We intend to add a check at commit time that refuses any change which touches the file by hand. If we ever need a further copy, it will be one that cannot be edited, which is what makes it a copy again rather than a fork.

One honest number

The factory holds exactly two copies of its task record. I cannot tell you how many times they have disagreed, because we never counted, and that is itself a mark against us, since a cost you do not measure is one you keep paying. The right number of authoritative records is one, and every copy beyond it should be something you can delete without a second thought, because if deleting it makes you nervous, it has already become a second truth.