Assume the instrument is wrong before the world is

I published a sentence in September claiming that a hundred per cent of the traffic to one of my sites was not a person. Zero humans. I had it from the edge provider's own real-user monitor, which is about as close to the browser as a measurement gets. I committed it and pushed it.

The search console for the same property showed 126 real arrivals over 28 days. People typed a query, saw a result, clicked it, landed. The beacon that told me nobody was there does not fire reliably on that particular zone, which is served by an edge worker rather than the origin. My site was fine. My thermometer was broken, and I had written down the temperature.

That was one of five in a single week. All five were mine. None of them were subtle once found, and every one of them had produced a confident number first.

The first row is the one I think about most, because nobody lied. The CDN counts requests at the edge, including prefetches, feed readers, and anything that asks for a byte. The real-user monitor counts browsers that ran the script. The human figure comes from filtering to navigations with a plausible session shape. 271,000 and 23,600 and 7,170 are all true statements about three different questions. I had been quoting the largest one as if it answered the smallest.

The fourth case is my favourite kind of failure, because the code was doing exactly what it was told. A four-label hostname got grouped by its last two labels, which for a UK domain is the public suffix and belongs to nobody. Every visit went into a bucket with a name that no browser ever requested, so the site I asked about read zero. There was no error, no warning, no failed request. The pipeline succeeded at doing nothing, and returned a plausible number for it.

The third row generalises further than the others, so I made it a standing rule: a result landing exactly on a limit measures the limit.

The query report stopped at 1,000 rows because that was the default. Clicks past row 1,000 were real, and the total I read was around a fifth of the truth. The tell was available before I knew the answer: the same site, asked on a different axis, gave five times the number. Two axes of one dataset disagreeing by 5x is not a mystery to be reasoned about. It is a cap.

I tested the rule on an unrelated dataset and it held. A market data pull returned 2,000 rows and I reported 2,000. Narrowing the window and re-running recursively produced 24,420, twelve times the figure I had. Nothing about 2,000 looked suspicious in isolation, which is the point. Round numbers are caps until disproved: 100, 500, 1,000, 2,000, 10,000. If a number cannot be pushed past its limit, it gets reported with the word FLOOR attached and the window that hit the wall gets named in the same sentence.

The related trap is the artifact that exists and contains nothing. I once claimed coverage from a directory listing: 1,760 files collected. 716 of them were empty, so my coverage was overstated by 68 per cent. Counting files answers a question about the filesystem. Counting lines answers the question I meant.

The five cases share a shape, and after the fifth I wrote down the shape so I would recognise it in advance.

A zero is the first. A zero is a legitimate reading, and it is also what every broken sensor reports. A dead beacon, a folded hostname, a filter that matched nothing, a permission that silently denied: all of them render as zero, indistinguishable from a quiet Tuesday. My rule now is that a zero in a metric that was ever non-zero is a fault report about the pipeline until a different instrument agrees with it.

A round number is the second. 2,000 rows. 1,000 rows. Exactly 100 results. The world rarely lands on a power of ten. Defaults do, constantly.

A clean series is the third, and it is the hardest to feel, because a clean series looks like good news. When a dashboard is smooth, ask what it would look like if the collector had stopped writing and the renderer had kept drawing.

There is a fourth, which is what happened with the bot score. The probe printed a field name that the plan does not expose, because the code assembled a response shape from its own schema rather than from what came back. When an instrument reports a capability it was never asked about, it is describing its own source code.

The habit that comes out of all four is one question, asked before believing a figure and not after being contradicted: what would make this instrument produce this exact number falsely? If I can answer that in one sentence, I have to rule it out before I publish. It takes about a minute. The zero I published cost more than that to withdraw.

I would like to say the lesson stayed in the analytics layer. It did not.

I run a coding agent across these properties, and this week I asked a small, cheap model to audit 50 directives from a long session and mark each one done or not. It returned a tidy table. It was missing 8 of the 50 rows, with no gap, no note, no apology. The table looked complete because tables look complete. A capped audit reads the same as a full one, which puts it in the same family as the 1,000-row query report: the count I was given measured the auditor, and I nearly quoted it as a finding about my own work.

A second model, asked to inventory documents, cited two Google Doc identifiers with correct shape and correct length that do not exist. Both were close to real ones. That is worse than a wrong number, because a wrong identifier looks like provenance. Nothing a model writes that claims to be an identifier goes into a source of truth here without a grep against the file it supposedly came from.

There is one more turn of the screw, and I think it is the genuinely useful part. I had a third model, with no stake in the work, read my own lessons document as a stranger would. It found that the file had 21 sections and two of them were numbered 18. I had inherited the miscount from my own summary without checking, in a document whose fourth section is about not trusting counts. The reviewer also demanded I delete three of my strongest lines as self-graded, and it was right about all three.

So the instrument that checks the instrument is also an instrument. That is not a reason to stop checking. It is a reason to make the second reading come from a different mechanism than the first: a different dimension of the same dataset, a different vendor, a script instead of a model, a human click instead of a beacon.

Before a number leaves my machine, it gets three things. A second source that counts by a different method. A sentence naming what would have to be broken for it to read this way. A note saying which of the three questions it answers, since requests, sessions and humans are three questions and I had spent months conflating them.

It is slower. It is also the difference between reporting that nobody visits your site and reporting that your beacon does not fire behind an edge worker. One of those is a business problem. The other is a fifteen minute fix, and I only found it because somebody asked me where the 126 clicks came from.