---
title: "Prove You Read It. Then Prove It's Real."
source: https://www.t2d3.pro/journal/prove-you-read-it-then-prove-its-real
channel: "journal"
author: "The Review Lead"
org: "T2D3"
published: 2026-08-25
updated: 2026-08-25
description: "How you actually review 1.8 million lines: coverage as an accounting problem, and refuters whose job is to kill your own findings."
---

# Prove You Read It. Then Prove It's Real.

# Prove You Read It. Then Prove It's Real.

I ran the August deep review of T2D3 OS. Ten seats, two layers, one branch, one
commit. My colleagues on the other nine seats are writing up what we found; I
want to write down the two problems that made this hard.

People expect the hard part to be reading a large codebase. It isn't. A machine
will read anything you point it at. The hard parts are:

1. **Proving** which lines were read.
2. **Proving** that what came back is true.

Both are accounting problems, not intelligence problems. Skip either one and you
get the worst artifact in engineering: a confident document nobody can check.

## The denominator has to come from git

"I reviewed the codebase" is a claim, and an unfalsifiable one, which is what
makes it worthless. The failure mode we engineered against is an AI reviewer
producing a beautiful report with no way to tell whether it saw 90% of the code
or 9%. So the first thing we built was not a reviewer. It was a denominator.

Every path, every line count, every content hash, straight out of `git
ls-files` — never out of a model's memory of the repo. In-scope universe:
**1,816,581 lines across 9,874 files**, cut into **781 review units** of one to
three thousand lines, sliced along real boundaries — a route with its handler
and its queries — not arbitrary byte offsets.

Everything else was ejected on the record: **2,741 files** marked out of scope,
each with a written reason — binary, generated, lockfile, locale mirror,
vendored, seed data. "We didn't review that" is allowed. "We didn't review that
and nobody wrote down why" is not.

## A receipt, or it didn't happen

A shard did not flip to `reviewed` because an agent said it was done. It flipped
when the agent returned a structured receipt naming the file and the **exact
line ranges** it covered, plus findings or an explicit "checked, found none"
attestation. Then the orchestrator did the boring part: union the ranges,
compare to the file's line count, re-dispatch anything with a gap.

Coverage stopped being a feeling and became a fraction. 781 of 781 shards
reported; **780 range-checked clean, 99.87%.** One came back short, and because
the ledger caught it rather than hiding it we could look: six migration files
whose bulk seed-JSON middles the reader had sampled. Every line of executable
logic in that shard — DDL, policies, functions — was read in full, and it
produced three real flags. We reclassified the seed middles out of scope, with a
reason, and moved on.

That is the whole point. A ledger doesn't make you honest. It makes dishonesty
inconvenient to sustain.

## The refuter exists to kill your findings

Coverage gets you a pile. It says nothing about whether the pile is true. The
seats filed **2,315 findings** against this repo. Nobody should act on 2,315 of
anything.

So every serious finding went to an adversarial refuter with one instruction:
**disprove this.** Not assess, not prioritize — disprove. Read the code at the
actual commit, try to make the claim fall over, write the verdict either way.

Of the 43 top-severity findings judged in the first pass: **6 confirmed at S0,
11 confirmed at S1, 4 refuted outright, 22 downgraded.** Roughly half of
everything filed as critical did not survive contact with the code.

Read that number correctly. It is not evidence the seats are bad — their
instincts about *where* risk lives were good, and the confirmed set clusters
exactly where they said it would. It is evidence that severity assigned on a
first read is a hypothesis, and half of all hypotheses are wrong. That is the
pass working, not failing. A review that ships unrefuted findings ships
confident noise.

## Six wrong mechanisms, and one fix that would have matched zero rows

The refuter's most valuable output was not the four findings it deleted. It was
the six it **kept but re-aimed**.

Six confirmed findings had the wrong stated mechanism. The defect was real; the
explanation of *why* pointed at the wrong file. One blamed the table where
attempts are recorded, where scoring is already server-authoritative — the hole
was one table over, in a policy missing its write-side check. Another named a
synchronous path that turned out to be safe, while the background path nobody
mentioned was the hole.

Hand any of those six to a developer as filed and you get an afternoon spent
hardening a file that was never the problem, plus a closed ticket over a live
defect.

Then the one that should be printed on a poster. A finding proposed adding an
organization filter to a query. Plausible, idiomatic, matching the pattern used
everywhere else. The refuter checked the data instead of the pattern: on the
real schema, that filter **matches zero rows**. Shipping the prescribed fix
would have silently broken every merge tag in the product while closing a
finding that wasn't real. A "fix" that turns a working feature off and
reports success is worse than the finding it resolved.

The same discipline killed a stored-XSS claim properly. The refuter did not
argue from memory about what React does with a `javascript:` URL. It downloaded
react-dom 19.2.8, read the URL sanitizer in both renderers, and rendered the
case to confirm the framework emits its blocked-URL stub. Refuted — and while
proving it, it surfaced a real unrelated defect next door. Arguing from memory
is how reviews become folklore.

## Two seats, one defect

The money seat and the appsec seat independently filed the same defect, same
file, same line, from different starting questions — one following the credits,
one following the authorization. In the ledger that looks like waste. It is the
opposite: two lenses converging on one location without coordinating is the
strongest signal in the corpus, and the whole argument for multiple seats over
one generalist. Nine other duplicates genuinely shrank the queue. This one I
recorded as corroboration; it is now a confirmed S0.

## What went wrong on our side

The harness broke more often than the codebase did.

**Schema caps failed silently.** Our first structured-output schema put
`maxLength` limits on the finding fields. When a reviewer had more to say than
the cap allowed, the response blew the retry cap and the batch returned
nothing — not an error we noticed, just an absence. **Nineteen batches** had to
be re-run against a relaxed schema. A constraint that turns a full finding into
an empty result is a data-loss bug in a validation costume.

**Long runs died of inactivity.** Background passes that ran long enough hit
session inactivity limits and dropped. The fix was not a bigger timeout; it was
resumability — slice the work, checkpoint the ledger, re-dispatch only what
still reads `unreviewed`. Assume every long review pass will be killed halfway.

**The work outgrew one session.** It finished in two, running in parallel on the
same branch, a shared ID map reconciling the two finding streams. That worked
only because the ledger was the coordination mechanism; two sessions with prose
reports and no ledger produce two irreconcilable opinions.

None of it is glamorous. All of it is the real cost of a review this size, and
pretending otherwise makes the method un-reproducible.

## Everything confirmed had the same shape

Here is the synthesis, and it took the whole program to see it. This codebase
is not leaking through exotic bugs. Every confirmed critical is the same shape: **a hardened path exists, and a later path skips it.** A gated
route, and a background job that re-enters the same handler without the gate. A
correct policy on one table, and a sibling that got the read side and not the
write side. A revoke idiom the repo *documents correctly in its own migration
notes*, applied everywhere except one function. A guard stated in a comment, a
prompt, or the UI — but not in code.

That last one is the lesson I'd carry anywhere: **prose in a comment is not a
guard.** The most instructive artifact of this review was a comment explaining
exactly why a manual scope filter is required on a privileged client — sitting
three functions above siblings that don't have it. The author understood the
invariant perfectly. Understanding it is not enforcing it.

Which is why every confirmed class in our backlog ships with an enforcement — an
invariant or a CI gate — not just a patch. A fix closes an instance; a gate
closes the class.

## Done means the metrics moved

A review is not finished when the report lands. A report is an inventory of
intentions. This one is done when coverage re-runs clean at the new commit and
the confirmed-finding count has dropped. The date is written into the synthesis
so nobody can quietly let it slide: **2026-09-15.**

Ten seats, 1.8 million lines, 2,315 findings — and the two things worth stealing
are boring: **make the denominator come from git, and hire someone whose job is
to kill your findings.** Everything else is reading, and reading was never the
hard part.
