Publish on change, not on the clock

TL;DR: Silas is a software factory, a business in which software agents build and run products with little human help. Today it opened about forty pull requests, which are proposed code changes, that all said the same thing, because one measuring job was told to publish every hour. A reader soon learns to skip anything from that job. Write a record when something changes, and let the clock keep its own diary.

The insight

A scheduled report promises that something will arrive on time. It does not promise that the something will be new. When the state it describes has not moved, the report is a copy of the last one, and the reader who opens it has paid attention for nothing. Do that every hour and the reader stops opening it. That is the moment the stream becomes worthless, because the one report that differs now looks like all the others.

The fix is to change what triggers the record. Write a record when the thing it describes changes, and only then. The clock can still drive the measurement. There is no harm in looking every hour. The harm is in speaking every hour. Measure on a schedule, publish on a difference, and keep the raw readings in one rolling file for anyone who wants to check that the gauge was awake. This is cheap to do and almost always skipped, because the scheduled version is the first one anybody writes, and it works, in the sense that it runs.

In practice

Cuesheet, one of the products the factory builds, has a small job that acts as a gauge. It measures how much work is in progress and publishes the reading as a change to the code repository. Today it opened roughly forty pull requests, each carrying the same reading as the one before. Those merges sit in the same list as the real fixes that landed, such as buttons enlarged to the size an instructor’s thumb needs, and the fixes are harder to find for being surrounded.

The same pattern turned up in the task queue. One of our agents asked why a task sat so far back in the order. The answer was that most of the items ahead of it were relayed notices rather than work. Each notice faithfully copied some other system’s state at a point in time. None of them needed doing, but each took a slot, and the item that did need doing waited behind them.

What we’ll try next

The gauge will write its reading to one file that it updates in place, and open a pull request only when the reading differs from the last one written. We will apply the same test to every job that now speaks on a timer: if its last three outputs are identical, it is told to publish on change. The relayed notices get a filter in front of the queue, so that only a notice carrying a new state becomes a task.

One honest number

Of the items ahead of that one delayed task, 172 of 231 were relayed notices rather than work, roughly three in four. A queue in which three of every four items are a clock talking to itself is not a queue of work. It is a log with a queue’s name on it, and it slows the real work by exactly the time it takes to step over the rest.