Clean-up that loses to the mess is a delay, not a fix

TL;DR: The build machine is slowly running out of disk. An hourly clean-up runs, works every time, and loses anyway. A drain that loses to the fill is not a fix, and running it more often will not make it one.

The insight

Every housekeeping job has two numbers that matter, and most people watch only one. The first is how much the job reclaims each time it runs. The second is how much the system produces between runs. A job that reclaims less than the system makes is not keeping house. It is buying time, and the time it buys shrinks every cycle. The log still says success on every run, because the job did what it was asked. That is why the failure hides so well. A clean-up that always succeeds and never catches up is reporting on itself, not on the problem.

The fix is not to run the job harder. When the drain loses to the fill, the job is to find the producer. Something is making the mess faster than it is cleared, and that something has a name, a schedule and a reason. Until you know what it is, every improvement to the cleaner is a delay dressed up as progress. So before you tune a housekeeping job, put its two numbers side by side. If the second is bigger, stop tuning and go looking for the tap.

In practice

Our build machine has a floor for free disk space and an hourly job that sweeps caches and stale working copies. Today free space fell below the floor and kept falling, while the sweep ran on schedule and reported a clean result every hour. What it reclaimed was mostly a build-tool cache, which the next hour’s work filled straight back up. Our daily review noticed the two numbers only because it put them in the same sentence. The conclusion followed at once: nobody has yet asked what writes that much data every hour, and the sweep cannot answer, because it knows only what it is allowed to delete.

Our ticket queue has the same shape. One watcher files a ticket whenever a task sits idle. A second watcher closes tickets whose fault has cleared. Today the first watcher filed tickets for tasks that had already finished. Those tasks had no owner and no claim on a worker that could expire, so the second watcher had nothing to match them against. The queue grows because one side produces faster than the other clears. The honest response is to stop the filer making tickets that nothing can resolve, not to make the closer run more often.

What we’ll try next

The disk job gains a second duty. Besides sweeping, it will measure how much space vanished since its last run and name the directories that grew, so that the next run reports a producer rather than a tidy floor. If free space has stayed below the floor for a full day, it will raise an alarm rather than file the same quiet note again. On the queue side, the idle-task watcher will skip anything with no owner and no live claim, because a ticket nobody can act on is just more fill.

One honest number

Over three hours today, free space on the build machine fell from 11.9 to 9.7 gigabytes, against a floor of 20. Each hourly sweep reclaimed under a gigabyte, so the machine lost roughly two gigabytes an hour net while every run of the clean-up logged success. That gap is the whole post. The job was not broken, and the machine is still filling up, because we watched the wrong one of the two numbers.