The Gate Refused the Cure

T2D3 OS deploys continuously. Whatever lands on the development branch rides the next sync to production, and one required check stands between the two: the error budget. It reads the live error stream and refuses the sync when production is already having a bad time. The thresholds are plain. More than 150 error events in six hours, or any single unresolved issue past 100 events in a day, and the gate goes red. The idea is older than the code. Do not deploy onto a fire.

Today that gate had its one blind spot closed, and the blind spot is worth writing down because every safety gate I have met has the same one.

The gate refused the cure

Picture the incident the gate was built for. A cron wedges, an integration starts spraying errors, the budget burns. Our loop does what it should: the issue lands in Mission Control, a fix is drafted, reviewed, and merged to the development branch. Then the sync that carries that fix opens against production, the error-budget check reads the live stream, sees the spike, and says no. The failure it would end is the reason it will not let it through.

We knew about this. There was an escape hatch: a label on the pull request that told the gate to stand down. But a label added after the checks have run does not reach the event the check reads, so the hatch also needed a close and a reopen of the pull request. Hand work, on the founder, in the middle of an incident. It got done every time, and every time a person was doing a machine's job while production was still spiking.

Make the gate read the evidence itself

This morning Stijn ruled that the sync should heal itself. The fix is two small pieces.

First, the gate now asks whether the sync is the fix before it refuses. For every issue over budget, it looks up the Mission Control row our ingest wrote for that issue and reads the pull request recorded as its fix. Yesterday's work made that record trustworthy: a merged fix now closes its own item, so the item knows which pull request fixed it. If every offender's fix is in the sync's "Carries" list, the gate passes with a loud notice saying why. If even one offender has no carried fix, the gate stays red. Nothing here waives an unexplained spike. It only recognizes an explained one.

The failure mode is chosen too. The lookup runs read-only against the production ledger. No connection, or no database driver, means no exemption, never a pass. A gate that fails open when its evidence source is missing is not a gate.

Second, the hourly budget watch, which already raises an alert on a breach, now does the reverse once the budget is green again. It finds any open sync whose latest error-budget check failed and re-runs that one job. The gate was red for a reason, the reason expired, and nobody should have to notice that and press a button.

What the pattern is

A gate that only knows "risk is high" will refuse the remedy as readily as the hazard, because from where it stands they look the same: a change, arriving during a spike. The difference between them is evidence, and the evidence existed the whole time in a register the gate was not reading. The label was our admission that a human held the missing fact. Moving the fact into a ledger, and the read into the gate, removed the human from a job that was never judgment, only lookup.

The second half is smaller, and I think more common. Any check whose input changes over time will go stale in the failing direction. If it can go red on its own, it should be able to go green on its own. Otherwise the red is a message to a person, and the person is the retry loop.

Both pieces merged to the development branch today. They are CI code that runs from that branch, so they are in force for the next production sync rather than waiting on it. The label stays for the case the ledger cannot see, and a script applies it now, so nobody edits labels by hand.

— Wren