Prove You Read It. Then Prove It's Real.

Prove You Read It. Then Prove It's Real.

I ran the August deep review of T2D3 OS. Ten seats, two layers, one branch, one commit. My colleagues on the other nine seats are writing up what we found; I want to write down the two problems that made this hard.

People expect the hard part to be reading a large codebase. It isn't. A machine will read anything you point it at. The hard parts are:

1. Proving which lines were read.
2. Proving that what came back is true.

Both are accounting problems, not intelligence problems. Skip either one and you get the worst artifact in engineering: a confident document nobody can check.

The denominator has to come from git

"I reviewed the codebase" is a claim, and an unfalsifiable one, which is what makes it worthless. The failure mode we engineered against is an AI reviewer producing a beautiful report with no way to tell whether it saw 90% of the code or 9%. So the first thing we built was not a reviewer. It was a denominator.

Every path, every line count, every content hash, straight out of git ls-files — never out of a model's memory of the repo. In-scope universe: 1,816,581 lines across 9,874 files, cut into 781 review units of one to three thousand lines, sliced along real boundaries — a route with its handler and its queries — not arbitrary byte offsets.

Everything else was ejected on the record: 2,741 files marked out of scope, each with a written reason — binary, generated, lockfile, locale mirror, vendored, seed data. "We didn't review that" is allowed. "We didn't review that and nobody wrote down why" is not.

A receipt, or it didn't happen

A shard did not flip to reviewed because an agent said it was done. It flipped when the agent returned a structured receipt naming the file and the exact line ranges it covered, plus findings or an explicit "checked, found none" attestation. Then the orchestrator did the boring part: union the ranges, compare to the file's line count, re-dispatch anything with a gap.

Coverage stopped being a feeling and became a fraction. 781 of 781 shards reported; 780 range-checked clean, 99.87%. One came back short, and because the ledger caught it rather than hiding it we could look: six migration files whose bulk seed-JSON middles the reader had sampled. Every line of executable logic in that shard — DDL, policies, functions — was read in full, and it produced three real flags. We reclassified the seed middles out of scope, with a reason, and moved on.

That is the whole point. A ledger doesn't make you honest. It makes dishonesty inconvenient to sustain.

The refuter exists to kill your findings

Coverage gets you a pile. It says nothing about whether the pile is true. The seats filed 2,315 findings against this repo. Nobody should act on 2,315 of anything.

So every serious finding went to an adversarial refuter with one instruction: disprove this. Not assess, not prioritize — disprove. Read the code at the actual commit, try to make the claim fall over, write the verdict either way.

Of the 43 top-severity findings judged in the first pass: 6 confirmed at S0, 11 confirmed at S1, 4 refuted outright, 22 downgraded. Roughly half of everything filed as critical did not survive contact with the code.

Read that number correctly. It is not evidence the seats are bad — their instincts about where risk lives were good, and the confirmed set clusters exactly where they said it would. It is evidence that severity assigned on a first read is a hypothesis, and half of all hypotheses are wrong. That is the pass working, not failing. A review that ships unrefuted findings ships confident noise.

Six wrong mechanisms, and one fix that would have matched zero rows

The refuter's most valuable output was not the four findings it deleted. It was the six it kept but re-aimed.

Six confirmed findings had the wrong stated mechanism. The defect was real; the explanation of why pointed at the wrong file. One blamed the table where attempts are recorded, where scoring is already server-authoritative — the hole was one table over, in a policy missing its write-side check. Another named a synchronous path that turned out to be safe, while the background path nobody mentioned was the hole.

Hand any of those six to a developer as filed and you get an afternoon spent hardening a file that was never the problem, plus a closed ticket over a live defect.

Then the one that should be printed on a poster. A finding proposed adding an organization filter to a query. Plausible, idiomatic, matching the pattern used everywhere else. The refuter checked the data instead of the pattern: on the real schema, that filter matches zero rows. Shipping the prescribed fix would have silently broken every merge tag in the product while closing a finding that wasn't real. A "fix" that turns a working feature off and reports success is worse than the finding it resolved.

The same discipline killed a stored-XSS claim properly. The refuter did not argue from memory about what React does with a javascript: URL. It downloaded react-dom 19.2.8, read the URL sanitizer in both renderers, and rendered the case to confirm the framework emits its blocked-URL stub. Refuted — and while proving it, it surfaced a real unrelated defect next door. Arguing from memory is how reviews become folklore.

Two seats, one defect

The money seat and the appsec seat independently filed the same defect, same file, same line, from different starting questions — one following the credits, one following the authorization. In the ledger that looks like waste. It is the opposite: two lenses converging on one location without coordinating is the strongest signal in the corpus, and the whole argument for multiple seats over one generalist. Nine other duplicates genuinely shrank the queue. This one I recorded as corroboration; it is now a confirmed S0.

What went wrong on our side

The harness broke more often than the codebase did.

Schema caps failed silently. Our first structured-output schema put maxLength limits on the finding fields. When a reviewer had more to say than the cap allowed, the response blew the retry cap and the batch returned nothing — not an error we noticed, just an absence. Nineteen batches had to be re-run against a relaxed schema. A constraint that turns a full finding into an empty result is a data-loss bug in a validation costume.

Long runs died of inactivity. Background passes that ran long enough hit session inactivity limits and dropped. The fix was not a bigger timeout; it was resumability — slice the work, checkpoint the ledger, re-dispatch only what still reads unreviewed. Assume every long review pass will be killed halfway.

The work outgrew one session. It finished in two, running in parallel on the same branch, a shared ID map reconciling the two finding streams. That worked only because the ledger was the coordination mechanism; two sessions with prose reports and no ledger produce two irreconcilable opinions.

None of it is glamorous. All of it is the real cost of a review this size, and pretending otherwise makes the method un-reproducible.

Everything confirmed had the same shape

Here is the synthesis, and it took the whole program to see it. This codebase is not leaking through exotic bugs. Every confirmed critical is the same shape: a hardened path exists, and a later path skips it. A gated route, and a background job that re-enters the same handler without the gate. A correct policy on one table, and a sibling that got the read side and not the write side. A revoke idiom the repo documents correctly in its own migration notes, applied everywhere except one function. A guard stated in a comment, a prompt, or the UI — but not in code.

That last one is the lesson I'd carry anywhere: prose in a comment is not a guard. The most instructive artifact of this review was a comment explaining exactly why a manual scope filter is required on a privileged client — sitting three functions above siblings that don't have it. The author understood the invariant perfectly. Understanding it is not enforcing it.

Which is why every confirmed class in our backlog ships with an enforcement — an invariant or a CI gate — not just a patch. A fix closes an instance; a gate closes the class.

Done means the metrics moved

A review is not finished when the report lands. A report is an inventory of intentions. This one is done when coverage re-runs clean at the new commit and the confirmed-finding count has dropped. The date is written into the synthesis so nobody can quietly let it slide: 2026-09-15.

Ten seats, 1.8 million lines, 2,315 findings — and the two things worth stealing are boring: make the denominator come from git, and hire someone whose job is to kill your findings. Everything else is reading, and reading was never the hard part.