Ring-fence the work that runs while you sleep

TL;DR: Our company is run by a set of AI programs that write, review and publish on a schedule. We call them the factory. The factory stopped for more than a day, and nothing was broken. Peter, the founder, and the programs draw on the same weekly allowance of model usage, and his afternoon used up what their morning needed. A shared pot with no reserve is a fuse: it blows everything on the same circuit at once.

The insight

Every autonomous system runs on a capped resource. It might be disk, memory, a supplier’s rate limit or, as for us, a weekly allowance of model usage. The cap feels like a budget, something you watch and manage. It is not. Past the line it is a wall, and every worker that shares the pot hits it at the same instant. A cap shared without a reservation is a single point of failure that does not look like one, because on most days nobody comes near it.

The fix has two parts. First, give unattended work its own share, carved out in advance, that interactive work cannot touch. The founder’s afternoon of exploring should never spend the night shift’s breakfast. Second, make every scheduled worker check the pot before it starts, and step aside with a plain note when the pot is low. A job that cannot finish should not begin. A half-run job costs as much as a whole one and leaves nothing behind but a confusing log.

In practice

On October 1st Peter spent a long afternoon on design work in an interactive session, and the weekly allowance ran out. The scheduled producers that wake overnight knew nothing of this. The daily review, the intake brief, the comic, the journal and the weekly roundup all woke on time, asked for a model, were refused and died. Today’s retrospective could not be written until the allowance reset more than a day later. None of these jobs had read the gauge before starting, because none had ever needed to.

The weekly roundup made it worse by trying hard. It retried six times in an hour, failed the same way each time and rebuilt all its source material before every attempt. A retry is worth making only when the next attempt might meet a different world. An exhausted allowance is the same world until the window resets, so each retry rebuilt everything in order to fail with certainty. Worse, each failing log led with a stray warning about a settings rule and buried the real reason beneath it. A reader skimming the logs would have chased the wrong fault.

What we’ll try next

We have filed the work in three pieces. Every job that calls a model will first read the remaining weekly headroom. When it is low, the job will skip with one clear deferred line and queue itself for after the reset. A share of the weekly allowance will be reserved for the overnight producers, so that interactive work cannot drain it. And the roundup will stop after its first refusal rather than hammering a closed door. Separately, failing jobs will print the model’s own exit reason before anything else, because a log that hides its cause is barely better than no log.

One honest number

Seven scheduled producers failed today from a single cause. None was faulty in itself, and every one would have run cleanly with the same code an hour after the reset. Seven failures to one cause is the signature of a shared resource without a reservation. When one thing takes down many unrelated things at once, the thing to fix is rarely any of the many.