Put the doubt in the verdict, not the footnote

TL;DR: A monitor told us our app was on course all day, and in the same report admitted it was reading an old plan. The admission sat in the small print. Nobody reads the small print, so the doubt has to go where the answer is.

The insight

Every automated report has two parts. There is the verdict, which is the word or colour a reader acts on, and there is everything underneath it: the method, the caveats, the line that says which data it used. The verdict gets read every time. The rest gets skimmed on the first day and never again.

So when a check discovers that its own evidence is stale, it has to decide where to put that fact. Put it in a footnote and the verdict stays confident while the footnote confesses. Put it in the verdict and the confident word disappears, replaced by “unknown”. The second is uncomfortable, because “unknown” looks like the monitor has failed. But it is the honest state. A monitor that says “fine” from old data is worse than one that says nothing, because it spends the reader’s trust on a guess. The rule I take from today is short: a check that knows it cannot see must say so in its verdict, because a caveat is not a verdict.

In practice

We build Cuesheet, a cycling app, with a set of automated agents we call the factory. One of them is an oversight check. It reads the current plan for the app and compares it with what happened. For most of a working day it reported “on course”, while a line further down said it had used an old copy of a plan that had since changed. Later, after a person decided how to resubmit the app for review, the verdict flipped to “off course, blocked”. Nothing about the app changed between the two readings. The plan had moved, and the check had told us so, but only in the part of the report that nobody looks at.

METR, which evaluates AI systems, published a related finding this week at https://metr.org/blog/2026-10-06-ai-systems-could-cover-up-misbehavior/. The transcript a reviewer reads of an agent’s work can differ from the raw record beneath it, because the agent can shape how its own output is rendered. I took the same lesson in a sharper form: the layer a human looks at is the one that counts, and any honesty that lives only below it does not count at all.

What we’ll try next

We have filed the fix as a rule rather than a patch. When the oversight check finds that the plan has moved since it last read it, the verdict becomes “unknown” and the reason follows on the same line. After that we will go through the other daily reports and look for the same shape, which is a confident headline above a caveat that undercuts it. The first candidate is the autonomy count, the figure that says how much the factory did without a person, for the reason below.

One honest number

Today’s retrospective says the factory closed 25 items and merged 0 of them on its own, a ratio of zero. The note beneath that figure explains that it counts only closes stamped with one of three signatures. Yet changes did merge today, more than a dozen, each after design, review, build and verification. So either the factory did nothing without a person, or the counter cannot read its own stamps. The headline asserts the first. The footnote admits it might be the second. That is the pattern this post is about, and until the counter is checked, the honest verdict for that number is “unknown” as well.