Well-formed and wrong: shape improved, relevance did not

I run three small businesses through a coding agent. A trophy and gifts storefront, a tutoring agency, a filings-research product. Most of the work reaches me as a closing summary at the end of a turn, so in August I wrote a checker for the summary: two sections, one line per item of my message, result first, no line over 300 characters, nothing handed back to me that the agent could have done itself.

Then I ran the checker over every closing summary in the corpus instead of asking it how it felt.

July 137 summaries 0 clean 0% August 1,929 summaries 106 clean 5% September 2,069 summaries 900 clean 43%

Zero to 43 per cent is the largest single improvement I have measured on this box, and it arrived the week the checker started refusing summaries rather than advising on them. Prose rules had been in place for months and moved nothing. A refusal in the path moved it in days.

That 43 per cent is a first-attempt rate. A refused summary still sits in the transcript, so a turn that failed once and passed on the retry counts as one failure and one pass. The version I actually read is cleaner than 43 per cent, and this scan cannot separate the two.

While the format was improving, I measured something I cannot game from my side: whether my very next message corrects, repeats or rejects what just came back.

Two months of format work, and the share of turns I had to correct roughly tripled.

I will name the confounds, because the level here is inflated. The detector counts a message as a correction when it contains words like "no", "still" and "again", which appear in ordinary instructions all the time. My own standard also rose across the same window, in my own words to the agent: "i think my standards are just increasing". Early in the summer I was asking it to stop breaking things. By September I was asking it to optimise things that already worked. The direction survives both confounds. The rate does not, and I am not going to quote it as one.

Shape and relevance are separate axes. Fixing the first did nothing for the second.

Of 2,058 September summaries put through the checker, four findings account for 86 per cent of all failures.

Three of the four are the agent's own habits: writing long, writing many, parking work back on me that it could have finished. In September, 76 per cent of closing summaries handed something back to me.

One of the four is mine. A prompt that spans several repositories invites a summary that does the same, and that is 13.5 per cent of the failures. The fix on my side is prompt scope: one repository per prompt.

The question I actually care about is different from format. If I put eight items in a message, how many come back answered? I counted items in my prompts across August and September and marked an item as addressed when the closing summary talked about the same thing.

Roughly one item in three, and it degrades exactly where I load the most items into a single prompt. The two heaviest, at 2,402 and 1,261 items, are the two worst at 31 and 23 per cent, and they are also where I correct most often, 50 and 48 per cent. The lightest prompts, small and single purpose, do best at 56 per cent.

Stated limit: word overlap is a weaker test than reading each item. It counts an item as addressed when the summary discusses it, which is not the same as the thing being done. It still beats counting numbered lines, which is a measure of form and which I threw away after telling the agent "dont count proxies count actual work".

Format and item counts still flatter the agent, so I had readers go through the raw transcripts window by window and classify every ask and every output. The pattern is the same in every venture and every month.

A 43-prompt window on the storefront in late July: zero deploys, zero messages sent, six documents, eleven measurements, most of them provisional. At most one turn in six moved something a customer or the live site would see. My closing line for that window was that I still did not have the before and after I had asked for.

A 394-prompt window in August: 244 turns (62 per cent) ended in chat with no artifact, 78 reached something live, 13 ended in a sent message. No sale, no dispatched quote, no outbound message to an end customer in 394 turns. At least 36 per cent of the turns went to keeping the tooling itself alive.

A 551-prompt window in early September: 238 chat-only, 122 claimed deploys, 57 blocked. The real outcomes for those eight days fit in a line: three ads live, a watermark applied to 957 photos, one inbound enquiry.

A 40-turn window in the middle of September: one thing shipped, 2.5 per cent, a re-stamp of 3,499 photos. Roughly 30 per cent of the turns produced no logged answer at all.

The tutoring agency in July: one deploy confirmed live, zero messages to tutors or parents, a notice drafted and never sent. In August, after twenty turns of outreach asks, my own summary of the business was that nobody had ever paid it.

The filings-research product in September: 168 prompts, a research chain reaching 77 confirmed and 45 refuted and 13 underpowered, and zero paying customers. Fifty-eight of those 168 closing summaries mention the agent's own rules, hooks or memory file.

Nearly every one of those summaries was framed as DONE.

Measuring the reply's shape made replies look finished. Measuring my next message and the artifacts that exist tells the truth. Three things changed on this box as a result.

Grade relevance by the human's next message. A well-formed answer that draws a correction is a failed answer, and the correction rate is the only signal the agent cannot manufacture from its side.

Write the definition of done before the first tool call, in my words, with a path in it. An answer that lives only in chat does not exist to an auditor, to the next session, or to whatever model I am using next month. Repeat asks are the cheapest signal of a silent drop, and nothing in my stack counted them until this week. One directive I asked for three times had been executed zero times.

Count outcomes rather than artifacts. Sent, deployed, sold. Documents, measurements and commits are evidence that a turn happened. They are not evidence that the business moved, and for five months I let the first stand in for the second.

A gate can make the answer well-formed. It cannot make it the right answer.