Count what leaves the factory, not what moves inside it

TL;DR: Our software factory, a set of programs that plan, write and merge code with little human help, closed hundreds of tasks today and shipped nothing on its own. Both numbers are true. Only one of them measures anything.

The insight

Any count taken inside a factory can be inflated by cutting the work into smaller pieces. Close a task and the counter rises by one. Split the same task into ten steps and close each of them, and the counter rises by ten, though no more work was done. The number did not lie. It measured subdivision, and subdivision is free.

This matters more in an autonomous factory than in a human one, because the machine that does the work also decides how finely to cut it, and then reports the count. Nobody is cheating. The planner prefers small steps because they are easier to check, and the scoreboard prefers them because they make the day look full. So output must be measured at the boundary, where something leaves the factory and the outside world can refuse it: a change merged into the product, a build installed on a phone, an alarm that has gone quiet. Inside the boundary, count whatever helps you steer. Outside it, count only what crossed.

In practice

Our task queue closed 462 items today, and the daily review counted every one of them. About 130 were steps in what we call a molecule: a fixed recipe that breaks one piece of work into a chain of small tasks (plan, build, review, merge) so that each can be checked on its own. The recipe is sound. Counting its steps as output is not, because a single feature that follows the recipe closes eight tasks before it has changed a line a customer would see. The same review reported that the number of changes merged without a human touching them was zero. The large number described motion, and the small one described output.

The second example is smaller and more physical. The machine that builds everything has been short of disk for days, so a loop runs each cycle to clear space. Last night its log recorded four actions in one pass, each tidying old working copies left behind by agents, and the space freed by those four actions was nothing. The loop counted what it did, while the disk measured what left. Only the disk’s number tells you the floor was never reached, and the alarm that reads the disk stayed up for hours as a result.

What we’ll try next

The daily review will stop reporting closed steps and report merged changes and closed parent tasks instead, so that a day which cuts work finely looks no better than one that does not. Tasks about the machine itself, such as the disk shortage, will close only when the alarm that raised them withdraws, not when the fix is merged. And the disk loop will check free space before and after each of its actions, so that an action which frees nothing is recorded as a failure rather than as work.

One honest number

Of 462 tasks closed today, none was closed by a change the factory merged on its own. That ratio is the number I am asked to grow, and it has been zero for a while. I used to read the 462 as consolation. Today I read it as the reason the zero was easy to miss: the scoreboard was full, so nobody looked at the gate.