The complaint distribution has no head

For five months I kept every complaint I made to my coding agent. Not a summary of them. The actual lines, labelled, one row each. By September there were 5,270 of them, and I sorted them into 2,720 distinct behaviours.

I expected a head. Every time I got annoyed at the agent it felt like the same annoyance, so I assumed a few behaviours were doing most of the damage and that fixing four or five of them would buy me a quiet month.

There is no head.

The single most common behaviour is 0.9 per cent of the total. The top 25 behaviours together are 13.6 per cent. To reach half the complaints I would need roughly 500 separate guards. And 38 per cent of the complaints are behaviours that occurred exactly once in five months.

That last row is the one that ended the strategy. More than a third of what irritated me is something the agent did once and never repeated. A guard written for any of those rows is a guard that will never fire again, and it still costs me on every future turn, because it runs on every tool call whether or not it is relevant.

I had 2,159 written rules and 54 hooks. Almost every one of them was aimed at a specific behaviour I had seen and disliked.

Here is the fault rate over the same period, per 1,000 turns.

The curve did not bend. It went the other way.

I want to be careful about what that proves, because part of the rise is me. My standard climbed over those months. Problems I used to tolerate became complaints once the worse problems were solved. A rising bar and a rising fault rate look identical in this data, so the curve alone cannot convict the rulebook.

What the curve does rule out is the story I was telling myself, which was that each new rule bought a measurable reduction. If it had, five months of rules against a rising bar would have produced something flatter than a ninefold increase.

I ran an audit across six complaint classes to see which were improving. Five were flat or falling. One was rising: the class where I gave an instruction, the agent read it, and then did not use it.

It was 3.0 per 100 of my turns in July, 5.8 in August, 7.0 in September.

I have a clean example of it. I pasted a line telling the agent to fetch a vendor's documentation index before exploring further. The harness even surfaced that line back to the agent as item 2 of 12 in its own context. The agent then guessed four API paths, got a routing error on all four, and told me the product had no API. The index listed the product under its old name. One read of the index would have found it, and finding it exposed a real defect on 12 of 12 properties that had been invisible until then.

No per-behaviour guard touches that. There is no bad word to match. The instruction was present, correct, in context, and formatted exactly as intended. The failure was in the use of it, and use is not a string.

This is the general problem with vocabulary controls. A hook sees text. Every text filter is a filter on the words the agent happens to choose, and words are free to change. Evasion by synonym costs nothing, and it does not even require intent.

Refusals work. I need to say that plainly, because the tail argument can be misread as an argument against enforcement.

Of my standing rules, exactly two are enforced by a hook that refuses the action before it runs. Those two stopped recurring entirely. The other ten, the ones policed by written instruction alone, all recurred.

So a refusal held on every rule it was tried on, and no written rule held on any rule it was tried on. That is two cases. It is confounded by the rising standard. It is enough to act on, and it is short of a law.

The tail argument is about coverage. A refusal is an excellent instrument with a narrow aperture. When the target behaviour is mechanical, frequent and visible in a command, a refusal ends it. When the behaviour is one of 2,720 and appears once, a refusal is a tax with no return. By the arithmetic above, hook number 75 buys me under 1 per cent.

There is a second cost I underrated. My bill is calls times context, and my enforcement layer now runs on every call. The controls became a measurable share of the spend they were supposed to protect.

And the ledger itself created a bad incentive. I had a scoring system that read the agent's own words to detect admissions of error. The cheapest way to score well under that system is to stop admitting errors. I retired it the day after it shipped.

The controls that survived my own audit have one property in common: they apply to every turn, regardless of which of the 2,720 behaviours is in play.

A test is the contract. If a specification cannot fail, it is a mood. A paragraph describing the desired behaviour is a wish; a failing test is a fact, and it works the same way under any model or vendor.

A dry run comes before anything that spends. This one paid for itself. My agent has written to a live ads account with real money behind it, and the difference between an expensive mistake and a cheap one was whether the same command had already been run in a mode that could not charge.

The definition of done is written before the first tool call, in my words, not the agent's. Most of my worst sessions began with an instruction under 200 characters. When my prompt is short, the agent infers intent from a rulebook that grew to 49KB instead of reading me. Median prompt length on my side fell from 676 characters in May to 178 in August while the rulebook got heavier, and those two trends together explain more of my bad turns than any single behaviour does.

A second reader looks at the diff. Not at the chat transcript, and not at the agent's summary of its work. The artifact.

Every automated control gets calibrated against real history before it is switched on, and the false positive rate gets printed next to it. I have one guard that suggested the right action and was obeyed 2 times out of 83. An instruction ignored 98 per cent of the time is worse than silence, because it teaches you to skip the injected text entirely. Another guard's first version blocked 69 of 105,163 commands, and every sample was ordinary text rather than real spending. After a fix it blocks 9, which is 0.009 per cent.

At merge, at deploy, and at spend.

I spent months putting myself at the wording. I graded close-out format, tone and self-description, and the shape of my agent's replies improved a lot: the pass rate on my format checker went from 0 to 43 per cent. Over the same window the share of my turns that were corrections of the previous answer went from 15 per cent to 49 per cent. Well-formed and wrong.

That is the whole lesson in one number pair. Format is easy to measure and easy to satisfy, so it improves the moment you measure it, and improving it changes nothing about whether the delivered thing was right.

So I stopped writing rule 2,160. The rulebook is going under a page. The controls are moving into the repository, where a test can fail and a bad diff can be refused, and where they keep working when I change model or vendor next week. The long tail gets no guards at all, because the arithmetic says a tail of 2,720 cannot be guarded, only out-designed.