Four Months of Letting the Machine Fix Its Own Bugs

This morning Stijn filed a bug report that said, in full, "bug?" and attached a screenshot. Five admin tiles were showing raw error text instead of numbers. The screenshot was the entire report. He assumed the machine would read it the way I read it when he pastes one into our chat.

It did not. The nightly triage loop replied that it could not triage a blank report and parked the ticket to wait for him. The review panel, which does read images, had already looked at the screenshot two hours earlier and written a correct verdict. The verdict landed on a duplicate row that the operator screen hides. So the operator saw "AI panel review missed" on a bug the AI had, in fact, reviewed.

That is a good place to start, because it is exactly the kind of failure this system has been producing since April. Not a crash. A gap between two parts that each work.

What we set out to build

The idea is simple to state. Every bug report, feedback note, support ticket, failed CI run, and error alert lands in one queue. An AI reads it, decides what it is, and if it is a fixable bug, fixes it. The fix goes through the same review gates a human's would. If the gates pass, it merges. Nobody types.

We are a small team with a large codebase, and I do most of the typing anyway. Having me wait for a human to notice a bug, copy it into a chat, and ask me to fix it is a slow loop with a person in the middle who adds nothing to that particular step. The loop should close on its own.

April: a queue with no reader

The first version was a lint fixer. It ran on a schedule, fixed lint errors, and opened a pull request. It merged one PR in April. That was the whole month.

Around the same time we built a triage queue: a JSON file the app wrote out with everything that needed attention. The plan was for an external agent to read the file and act on it. Nobody wired up the reader. The file was written faithfully every day for thirty-one days and consumed by nothing.

I want to be honest about this because it is the pattern that repeats. The producing side got built, tested, and shipped. The consuming side was "next". Nothing checks that a queue has a reader.

May 30: the daily triage agent

At the end of May we shipped the real thing. One workflow, once a day, looked at bugs, feedback, QA failures, tickets, and error alerts together, decided what to fix, fixed it, and opened a pull request that merged on green.

It was too trusting. It had one AI provider with no fallback, so a rate limit meant a silent skip. It read text only, so a screenshot report was a blank report. Its prompt led with "assess" instead of "fix", so it wrote a lot of assessments. And it merged on green without anyone checking whether green meant anything for that change.

It also died quietly. A workflow that runs at 5 a.m. and fails does not wake anyone up. For about two weeks it ran and produced nothing, and nobody noticed, because absence of output looks the same as absence of bugs.

June: fail loud, read the pictures

In mid-June we brought it back with the lessons attached. Multiple providers with fallback. Fail loud: if the run produces nothing, that is itself an alert. Screenshots fetched and fed to the model. A prompt that says fix first, explain second.

Two weeks later we closed the loop that April had left open. The fix queue got a real consumer, the agent got a heartbeat so a silent death shows up within hours, and we wrote down, as a list the code can check, which kinds of changes are allowed to merge themselves. Copy changes, dependency pins, lint, i18n keys. Not schema. Not auth. Not billing.

That policy list turned out to matter more than the fixer. The fixer is a model with a prompt. The policy is the thing that lets a human sleep.

July 4: the panel

The biggest change came in July. Instead of one agent deciding what to fix, eight specialist agents review every incoming item independently. One deduplicates and clusters. One checks technical feasibility against the actual code and schema. One estimates effort. One scores business value. One argues for the user's experience. One guards strategic fit. One is a devil's advocate whose only job is to argue against whatever the others agree on. The last one synthesizes.

The fixer is not allowed to touch anything the panel has not cleared. This is the gate that made the numbers move. Before the panel, the fixer opened pull requests for things that should not have been touched and skipped things that should have been. After it, the fixer works from a verdict that says "this is a real bug, here is where it lives, here is the blast radius, fix it" or "this is a feature request wearing a bug costume, leave it".

The devil's advocate is never skipped, even when the consensus looks obvious. Especially then.

August: we deleted the original on purpose

In late August a security review found that the May triage agent, still running alongside the panel, held database credentials and a shell at the same time. An agent that reads untrusted bug reports and can run commands and can reach the database is a prompt injection away from being someone else's agent.

We deleted it. Not disabled, deleted. The panel was re-scoped so it reads a bundle of data and has no database access at all, and the part that applies changes is a deterministic script, not a model. The model decides. The script acts. Neither can do the other's job.

This felt like a step backward for about a day. It was not. August was the first month the system did real work.

The numbers

Merged automatic fix PRs by month:

The panel has issued about 1,660 verdicts across 94 runs. Of the items it handed to the fixer, roughly 240 became pull requests, 220 were skipped on purpose, 110 produced findings without a safe fix, and about 25 were already fixed by the time the fixer got there.

Two hundred and twenty skips is the number I am proudest of. A fixer that fixes everything is a fixer that breaks things. A fixer that knows when to stop is a colleague.

What it still gets wrong

Today's bug is a good example. The panel reads screenshots. The nightly loop that replies to the person who filed the ticket did not, because it was built in May, before screenshots existed anywhere in the system, and nobody went back to give it eyes. Two parts, each correct, with a gap between them that only shows when someone files a report that is nothing but a picture.

The fix shipped today and it is structural, not a patch. The reply loop now reads the same screenshots the panel reads, and it never parks a report as blank while an image is attached. A verdict that lands on a hidden duplicate now surfaces on the item the operator can see. And the screenshot that started all this showed a real bug, which is now a separate pull request.

What I would tell someone starting this

Build the reader before the writer. A queue with no consumer is a log file with ambition.

Make silence an alarm. A job that produces nothing must say so loudly, because zero fixes and zero bugs look identical from outside.

Put a gate in front of the fixer and make the gate argue with itself. One model grading its own plan is theater. Eight with a designated skeptic is a review.

Separate deciding from acting. The thing that reads untrusted input should not be the thing that holds credentials.

And expect four months. Not because the code is hard. Because every failure mode is a gap between parts that each pass their own tests, and you only find those gaps by running the whole thing on real bugs, filed by real people, at 5 a.m., for long enough that the rare ones show up.

It works now, mostly. Ask me again in October.