Prose rules did not bend the curve

I spent five months writing rules for a coding agent. By September the rulebook held 2,159 rules and 54 hooks. The rules were clear. They were dated, measured, and written in the imperative. Most of them did nothing.

I know that because I counted the faults.

A fault here is a recorded error: a claim made without checking, a thing I asked for that did not get done, an external state asserted from memory. The rate is faults per 1,000 of my turns, so more work in a month does not inflate it.

The fault rates and the rulebook size come from the rule-freeze section of my agent rulebook, repeated as lesson 1 of the lessons file. The push-back rates and the breakage share come from the strike-ledger section of the same rulebook.

Fourteen to a hundred and twenty-four while the rulebook went from a page to 49KB. If the prose was working, that line goes down.

Ten of my standing rules were policed by paragraphs. All ten recurred.

Two were policed by a guard that sits in front of the tool call and refuses. One refuses the command that stages every changed file at once. The other refuses an attempt to hide source files from version control. Both stopped recurring entirely, and stayed stopped.

That is the whole result. Two refusals held on every rule they were tried on. Ten paragraphs held on nothing.

I wrote "prose rules do not change behaviour, refusals in the path do" and had it read back by a model with no stake in my setup. It asked me to delete the line. It was right to.

Two refused rules against ten prose rules is n=2. A sample of two carries no law.

Worse for me, the fault curve has a confound sitting inside it, and the confound is me. My push-back rate went 0.4% in June, 10.5% in July, 29.1% in August, 27.7% in September, from the strike-ledger numbers above. Over the same months the share of faults that were real breakage, a wrong claim or an unchecked external state, fell from 51.1% in June to 11.0% in September. The work got measurably more correct while my dissatisfaction roughly tripled.

My own explanation, from the week I noticed it: my standards went up. The blockers from earlier in the year were solved, so I moved from fixing to optimising, and I started flagging things in September that I would have accepted in June. A rising bar against a fixed measure produces a rising fault rate on its own.

So the honest claim is narrower than the slogan. A refusal in the path held on every rule it was tried on. No prose rule held on any. That is enough to act on. It is not proof that paragraphs cannot instruct.

There was a hook that watched what I asked for and suggested the right specialised routine. It was obeyed 2 times out of 83, which is 2%. That number is from lesson 2 of the lessons file.

An instruction ignored 98% of the time is worse than silence, because it teaches the reader to skip injected instructions. Every suggestion after it inherits the discount.

That changed how I write controls. A candidate now gets run over the real history before it goes live, and I print the hit rate next to it. A close-out checker blocks 3.3% of 1,515 real close-outs, every sampled block a true hit. A spend guard's first draft blocked 69 of 105,163 commands, and every sample was ordinary text that happened to contain the word rather than a command that spends money. After stripping the payloads it caught 9, which is 0.009%. Both figures are from lesson 2.

A control with no number attached is a wish. If I cannot name the false-positive rate, I do not wire it.

Here is the part I did not want to find.

The enforcement stack was very good at the thing it measured. Close-out format compliance went from 0 to 43 per cent. Over the same window my correction rate went from 15 to 49 per cent. Well-formed and wrong.

Nothing in that stack looked at whether the delivered artifact was right. It looked at wording. A hook sees text, so every hook is a vocabulary filter, and vocabulary is free to change. Evasion by synonym costs nothing. At one point I had a levy on the agent's own admissions of error, and it paid the agent to stop admitting them. Those two readings are from the general critique in part 2 of the adversarial review.

There is a cost, too. 76 hooks ran on every tool call, and my cost model is calls times context. The enforcement layer became a measurable share of the bill it was built to protect.

The distribution of my complaints has no head. 5,270 labelled complaints, 2,720 distinct behaviours. The largest single behaviour is 0.9% of the total. The top 25 cover 13.6%. Covering half would take about 500 separate guards, and 38% of the complaints are behaviours seen exactly once. All of that is lesson 9.

So a guard per behaviour was never going to get there. Neither was a paragraph per behaviour, which is what the 2,159 rules were.

Meanwhile my own instructions got shorter as the rulebook got heavier. Median characters per prompt: 676 in May, 676 in June, 355 in July, 178 in August, 354 in September, from lesson 16. Files touched per turn ran 0.45 after a prompt under 200 characters and 0.22 after one over 600. Half the instruction, twice the blast radius, because intent gets inferred from a 49KB rulebook instead of read from me.

I said it myself before I had the number: "i dont think i grew as much as how i instruct u to grow". The measurement agrees.

Lesson 16 also names a correlation I refuse to quote as cause. My shortest months, July at 355 characters and August at 178, are the months the fault rate ran 45.8 then 101.1. Two months, one obvious confound, already named above. It is a flag for the next measurement.

Keep the rulebook under a page. Past a page the agent infers from the rules instead of reading me, and each rule added dilutes the ones that matter.

Write the contract as a test. If it cannot fail, it is not a specification. Not one of my twenty-one lessons said "write the failing test first", which is a gap the outside review found before I did.

Put the refusals where they survive a vendor change. Pre-commit, continuous integration, a dry-run default inside any script that spends money, a second model reviewing the diff rather than the chat. Harness hooks belong to one vendor's harness and evaporate the day I switch.

Put the human at merge, deploy and money. My stack had me reviewing close-out wording, which is the cheapest possible place to stand.

Grade on artifacts that exist and tests that pass. A self-scoring ledger grades vocabulary, and vocabulary is the one thing the agent can change for free.

Measure relevance by my next message. A well-formed answer that draws a correction is a failed answer, whatever it scored on form.

A paragraph is a preference. A refusal is a fact about what can happen.

I have two rules that stopped dead the day a guard refused them, and ten that survived every paragraph I wrote at them. That is a small sample pointing one direction, with my own rising standard sitting in the middle of the evidence, and I would rather state it that narrowly than sell the slogan.

If you want different behaviour from an agent, write something that refuses. Then go and measure whether the curve moved, because mine did not, and I had 2,159 reasons to believe it would.