How to audit an AI-built codebase: the full program, every brief, every gate

This is the long version. It exists because the short version was read by people who then asked the obvious question: fine, but what exactly do I tell my CTO to run on Monday? Everything below is the answer. It is written so an engineering leader can hand it to one senior engineer and a budget, and get a trustworthy audit back in about six weeks, with the gates in place to keep it true afterwards.

Who this is for, and how to use it

If you run a software company and your team ships with Claude Code, Codex, Cursor or any of their relatives, the shape of your risk has changed. The code compiles, the tests pass, the pull request reads beautifully, and the defect is somewhere nobody looked, because nobody had to look. An agent wrote four hundred lines in ninety seconds and every one of them was plausible. Output went up by an order of magnitude. Human attention did not.

We built T2D3 OS almost entirely this way. As of this writing the codebase is around ten thousand TypeScript files, a thousand-plus API routes, five hundred-plus database tables and a thousand row-level security policies, and it was written by AI agents under human direction over eight months. We audited it four times in that period, each pass more rigorous than the last, and the program below is what survived. It found real cross-tenant holes before any customer did. It also produced, in its early form, thousands of findings that were roughly half wrong, and taught us that the audit itself has to be audited.

The guide is organised so it can be executed rather than admired:

• Part 1 explains the program in one page and gives the numbers that justify it.
• Parts 2 to 6 walk the five stages. Each stage has a purpose, the exact brief to paste into a model, the output format to demand, and the rules the orchestrator enforces.
• Part 7 covers the part most teams skip: auditing the gates and the audit machinery, because both fail silently.
• Part 8 is governance: how the findings become permanent checks, and how you keep them from regressing.
• Part 9 is the calendar, the checklist for the CTO, and the questions the CEO should ask when it is done.

Read Part 1 and Part 9 first if you are the CEO. Hand the whole thing to the CTO.

A note on cost. We deliberately do not quote our token spend in dollars. Model prices halve on a timescale shorter than this document's shelf life, and the honest answer is that the reading was always the cheap part. The expensive part was human judgment on what the reading produced, and that is the part this program is designed to protect.

---

Part 1 — The program on one page

Why one big audit fails

Ask a frontier model to review your codebase and it will find problems. That is not the hard part. The hard part is that roughly half of what it finds is wrong, and you cannot tell which half.

Our June pass filed 47 claims of the form "this table needs stricter access control" from a pattern-matching first read. An adversarial second agent, told only to disprove each one, killed 27 of them. Fifty-seven percent. Once we forced the finders to cite live evidence (the actual policy text, the actual query, the actual grant), the refute rate on new claims fell to about one percent. Same models, same codebase. The difference was whether the finder had to show its work.

Our August pass, the largest, put 449 findings filed at the two highest severities through refutation. Nine were refuted outright. Two hundred and forty-six were downgraded. Two hundred and seven survived at their filed severity. Half of everything filed as critical did not survive contact with the code, and that is the pass working. A review that ships unrefuted findings ships confident noise, and a developer handed confident noise spends an afternoon hardening the wrong file and closes a ticket over a live defect.

So the program is not "run a model over the code". It is a pipeline whose job is to convert volume into trust, stage by stage.

The five stages

Two rules sit across all five stages and are the reason the program works at all:

1. Enumerate, never sample. The denominator comes from git ls-files, not from a model's memory of the repository. Every unit of code is either reviewed with a receipt naming exact line ranges, or ejected with a written reason. "We didn't review that" is allowed. "We didn't review that and nobody wrote down why" is not.
2. Nothing reaches a human unrefuted. Every finding above the lowest severity goes to an agent whose only instruction is to prove it wrong, and a verdict survives only if every refuter confirms it. The refuter comes from a different model family than the finder, and it never sees the finder's reasoning.

What it found, in aggregate

Across four passes the program filed roughly nine thousand findings, confirmed a few hundred, fixed the top of the list within days of each pass, and promoted about a hundred and twenty defect classes into automated checks that now run on every pull request. The confirmed critical findings were, without exception, the same shape: a hardened path exists, and a later path skips it. A gated route and a background job that re-enters the same handler without the gate. A correct policy on one table and a sibling that got the read side but not the write side. A guard stated in a comment, a prompt, or the interface, but not in code.

That shape is why Stage 4 exists. A fix closes an instance. A gate closes the class.

---

Part 2 — Stage 0: Ground truth

You cannot audit what you have not counted, and you cannot rank what you have not measured. Stage 0 produces two artifacts before any model reads a line of code.

2.1 The denominator

The first thing to build is not a reviewer. It is a list.

# 1. Pin the commit. Every seat, every refuter, every re-run reads THIS tree.
git fetch origin main
PIN=$(git rev-parse origin/main)
echo "$PIN" > audit/PIN

# 2. A pristine, detached checkout with NO secrets in it.
#    Reviewers get read access to code, never to .env files or a database.
git worktree add --detach /tmp/audit-repo "$PIN"
test -z "$(find /tmp/audit-repo -maxdepth 2 -name '.env*')" || echo "STOP: secrets in the checkout"

# 3. The universe: every tracked file, its line count, its content hash.
cd /tmp/audit-repo
git ls-files -z | xargs -0 -I{} sh -c 'printf "%s\t%s\t%s\n" "$(git hash-object {})" "$(wc -l < {})" "{}"' \
  > ../audit/universe.tsv
wc -l ../audit/universe.tsv

Then you eject, on the record. Binary files, generated files, lockfiles, locale mirrors, vendored code, seed data. Each ejection carries a reason in a machine-readable file:

{"path":"package-lock.json","reason":"lockfile"}
{"path":"public/**","reason":"static assets, no logic"}
{"path":"messages/{de,fr,es,nl,pt,ja}.json","reason":"locale mirror of en.json; en.json is in scope"}
{"path":"src/lib/types/database.ts","reason":"generated from schema; schema is in scope"}

What remains is sliced into review units of one to three thousand lines, cut along real boundaries rather than byte offsets: a route with its handler and its queries; a migration with the policies it creates; a component with its hook. In our August pass the universe was 1.8 million lines across 9,874 files. 2,741 files were ejected with reasons. What remained became 781 review units. Those numbers are the audit's denominator, and every coverage claim later is a fraction over them.

Why this matters more than it looks. "I reviewed the codebase" is an unfalsifiable claim, which makes it worthless. The failure mode you are engineering against is an AI reviewer producing a beautiful report with no way to tell whether it saw 90% of the code or 9%. The list turns coverage from a feeling into a fraction.

2.2 The inventory by entity class

Alongside the file list, count the things that have blast radius. This is the table the seats are routed against and the table the synthesis is ranked against.

Take a copy of this table at every pass. The delta between passes is the single most useful trend line you will own. Between our June and August passes, in eight weeks, routes went from 515 to 906, tables from 158 to 480, policies from 661 to 920. That growth rate is why an audit that samples cannot keep up, and why the checks in Stage 4 have to be automatic.

2.3 The measurement pass

The second Stage 0 artifact is a sheet of numbers from production, taken before the seats read the code. This is the step that our own devil's advocate flagged in August as a structural flaw in our earlier passes: a static review ranks findings by how bad they look, and a hot path in production is invisible in source. We now run measurement first and make it a gate: no fix wave may start before its Stage 0 numbers exist.

Give one engineer read-only access to your telemetry and your production database and this brief:

BRIEF · STAGE 0 · MEASUREMENT PASS

You have read-only access to production telemetry and a read-only replica
of the production database. Produce a single markdown file, `audit/W0.md`,
with the following sections. Every number carries its query or dashboard
URL beside it so the next pass can reproduce it. Where a number cannot be
obtained, write "NOT MEASURABLE — <why>" rather than an estimate. Do not
propose fixes; this pass produces numbers.

1. FRONTEND (real-user monitoring, last 30 days, p75)
   - LCP, CLS, INP for the ten most-visited routes, ranked by visits
   - Pageload JS error rate per route
   - First-load JS bytes per route (run the bundle analyzer; if it will not
     run, say so — a broken analyzer is a finding)

2. BACKEND
   - p50 / p95 duration for the twenty slowest API transactions by volume
   - 5xx rate per route, last 30 days
   - The top ten database queries by total time (pg_stat_statements),
     each with its calling code path if you can identify it

3. DATABASE
   - Cache hit ratio; largest tables by size and by row count
   - Advisor output: unindexed foreign keys, unused indexes, tables with
     multiple permissive policies for the same role/action, policies
     re-evaluating auth functions per row
   - For every SECURITY DEFINER function: `has_function_privilege` for
     the anonymous and authenticated roles (yes/no per function)

4. SCHEDULED JOBS
   - Invocations per month, per job; failures per job; longest runtime
   - Any job that has not fired in its expected window

5. AI / LLM
   - Calls, tokens, and latency per call site, last 30 days
   - Provider error rate per call site
   - The share of LLM spend that reaches your call log. Compare the
     provider's billing console total against the sum in your own log.
     If they differ by more than 10%, the gap is a finding: something is
     calling the model outside your instrumentation.
   - Queue wait p95 for background AI jobs vs. the drain rate

6. THE DELTA
   - The entity-class inventory table, this pass vs. last pass.

Two of the numbers in that brief came from painful lessons and deserve emphasis. The first is the LLM spend reconciliation: in August we discovered that roughly two thirds of our provider bill flowed through paths that never wrote to our call log, because streaming endpoints and a few utility callers had grown up outside the dispatcher. The second is the bundle analyzer must actually run: ours had been silently vacuous for weeks because a build tool change had broken it, and a budget gate that cannot measure is a gate that always passes.

2.4 Stage 0 exit criteria

• audit/PIN exists and every later artifact names it.
• audit/universe.tsv and audit/ejected.jsonl exist; the sum of in-scope units plus ejected files equals the tracked-file count.
• audit/W0.md exists with a query or URL beside every number.
• The reviewers' checkout contains no .env file and no database credential.

---

Part 3 — Stage 1: The fan-out

Stage 1 is where the reading happens. Its output is a ledger, not a report.

3.1 The seats

A seat is a model session with one lens, one brief, and a slice of the universe routed to it. We run ten. Fewer is fine for a smaller codebase; what matters is that each seat has one question, because a generalist reviewer converges on the same twenty findings every time and misses the ones that need a specialist's priors.

Each seat is briefed to think like a named practitioner. This is not decoration. Naming the lens pulls a specific body of judgment out of the model, and it gives the refuter and the chair a shorthand for what the seat should and should not have caught.

Route review units to seats by path and by content. Every unit goes to at least the review lead; security-sensitive units (anything importing the service-role client, any policy, any route handler) go to seats 2 and 3 as well. Nothing goes to zero seats.

3.2 Batching

A seat does not receive its whole slice at once. It receives batches of three review units, in order, from a committed batch index. Three is the size at which a frontier model still reads every line rather than skimming; larger batches produced visibly shallower receipts in our runs. The index is a file, committed, so a killed run resumes from the ledger rather than from memory:

{
  "seat": "seat3_tenancy",
  "batches": [
    { "id": "s3-b001", "units": ["u0412", "u0413", "u0419"] },
    { "id": "s3-b002", "units": ["u0420", "u0421", "u0422"] }
  ]
}

Our August pass ran 617 batches across ten seats. Every batch either landed with a receipt or was re-dispatched. There is no third state.

3.3 The seat brief

This is the brief, verbatim, with the lens section swapped per seat. Paste it as the system prompt or the opening message of each seat session.

BRIEF · STAGE 1 · SPECIALIST SEAT

You are one of ten reviewers reading a codebase pinned at commit {PIN}.
Your checkout is at {REPO_PATH}. It is read-only and contains no secrets.
You cannot run the application or reach a database. Use rg, cat, sed -n,
find, and git log. The live schema is in db/schema-snapshot.sql.

YOUR LENS
{one of the lens paragraphs below}

THE RULES

1. Enumerate, never sample. For each file in the batch, read every line
   of executable logic. If a file has a bulk data middle (seed JSON, a
   generated table) you may skip the data rows, but you must say so in
   the receipt with the line range you skipped and why.

2. A claim is not evidence. Do not trust a comment, a doc, a rule name,
   a test name, or a PR description asserting that something is handled.
   Open the mechanism. A guard stated in a comment is not a guard.

3. Every finding needs a concrete failure. "This could be unsafe" is not
   a finding. "A caller with role X hitting route Y with body Z writes
   into another tenant's row, because line N filters on user_id alone"
   is a finding. If you cannot construct the inputs, set confidence to
   needs-verification and say exactly what you would need to check.

4. Quote the evidence. Every finding cites file and line and quotes the
   lines that prove it. A finding without a quote is discarded.

5. Bound yourself honestly. For every category your lens covers, either
   file findings or write "checked X, found none" naming the files. A
   report that lists only findings is unfalsifiable.

6. Severity, and do not inflate it:
   S0 = data loss, cross-tenant leak, or security breach reachable in
        production by a realistic actor
   S1 = user-visible breakage or silently wrong output
   S2 = a real defect with a narrow blast radius
   S3 = smell, maintainability, drift risk
   A single confirmed S0 is worth more than forty S3s, and every S0 and
   S1 you file will be sent to an adversary whose job is to disprove it.

7. Name the gate that failed. For each finding, say which existing check,
   test, lint or review step should have caught it and the mechanical
   reason it did not. If the repository has no such check, say
   "no gate exists". This field is often worth more than the finding.

8. Do not fix anything. Do not propose designs. Describe the defect and
   the smallest change that would close it.

OUTPUT
Return one JSON object per batch matching the RECEIPT schema below.
No prose outside the JSON.

The lens paragraphs:

LENS · APPLICATION SECURITY
You think like a penetration tester with the source. For every route
handler, background job and agent tool in the batch: who can call it,
what ids and free text does it accept, which of those reach a privileged
client or an external call, and where is the check that the caller may
act on THAT resource, not merely that the caller is authenticated. Trace
every caller-supplied id from the request body to the query that uses it.
Look for: privilege escalation via ids; injection into HTML, SQL, shell,
or LLM prompts; SSRF via user-supplied URLs; secrets on a command line;
CSRF-less OAuth state; anything under script-src 'unsafe-inline' fed user
data.

LENS · TENANCY AND DATA
You read policies as a set, not one at a time. For every table in the
batch that carries an organization or project column: list its write
policies and mark any whose only non-admin predicate is the actor's own
user id. For every SECURITY DEFINER function: who may execute it, and
does it do more than its name implies. For every use of the service-role
client: where is the tenant scope re-applied by hand, because the
database will not apply it. Agency operators span organizations in this
product; ask what an operator's write does at every site that resolves
the organization from the USER rather than from the RESOURCE.

LENS · BACKEND AND INFRASTRUCTURE
For every scheduled job and background handler: how is it authenticated,
what happens if it runs twice, what happens if it runs for an hour, who
finds out when it fails, and does it re-enter a handler whose route
version enforces a gate this path skips. For every write: is the error
inspected or discarded. For every queue: what is the drain rate versus
the arrival rate.

LENS · FRONTEND
For every client component: hooks before any conditional return; every
fetch in an effect aborted on cleanup; heavy libraries imported
dynamically; failures rendered as failures, not as empty states; loops
that keep running after unmount; save buttons that are wired. For every
page: a loading state, a metadata export, a single h1.

LENS · AGENTIC AI
For every agent, tool and autonomous loop: what data can it read, what
untrusted content can reach it, and what channel could carry data out.
All three together is the lethal trifecta; name it wherever it exists.
Every tool that mutates must leave a trail a human can audit. Every
destructive or outbound action must propose, never silently act. Every
loop must have a cap, a checkpoint, and an owner who is paged.

LENS · PROMPT ENGINEERING
For every LLM call site: where does the system prompt come from (a
resolved, versioned registry, or an inline string); what grounds the
generation (evidence the user can see, or nothing); is caller-supplied
free text guarded before it reaches the prompt; what caps truncate
silently; is there a feedback seam to capture human judgment on the
output; is the model the right tier for the task.

LENS · PERFORMANCE AND COST
Take the Stage 0 measurement sheet as your map. For every hot path it
names, find the code and explain the number. Then hunt the rest: N+1
query loops; select-star on heavy JSONB; polling with no reader;
uncached reads of static data; full re-fetches after one mutation;
first-load payloads that carry data the page never renders.

LENS · ARCHITECTURE
For every module boundary in the batch: what imports it, what it imports,
and whether a value-import cycle exists. For every utility: where else
in the repository does the same thing exist under another name. For
every "foundation" data structure: has it been bent to serve one
channel's needs. Name the seams where a change should be made once and
is currently made in N places.

LENS · MAINTAINABILITY
For every file: size against the limit; a test beside it or not; whether
a newcomer could find it by grepping for what it does. For every handler
that does real work: what asserts its behaviour, or is every guard on it
a grep. Hotspot analysis: files with high churn and high complexity are
where the next incident lives.

3.4 The receipt

A batch does not flip to reviewed because the seat said it was done. It flips when the seat returns a structured receipt naming the file and the exact line ranges covered, plus findings or an explicit attestation. Then the orchestrator does the boring part: union the ranges, compare to the file's line count, re-dispatch anything with a gap.

{
  "batch": "s3-b001",
  "seat": "seat3_tenancy",
  "pin": "3d124fee",
  "coverage": [
    { "unit": "u0412", "file": "db/migrations/2026…_workspace_members.sql", "lines_total": 212, "ranges_read": [[1, 212]] },
    { "unit": "u0413", "file": "src/app/api/workspaces/[id]/members/route.ts", "lines_total": 148, "ranges_read": [[1, 148]] },
    { "unit": "u0419", "file": "db/migrations/2026…_seed_templates.sql", "lines_total": 4210,
      "ranges_read": [[1, 88], [4102, 4210]], "skipped": [[89, 4101]], "skip_reason": "bulk seed JSON, no DDL or policy text" }
  ],
  "findings": [
    {
      "id": "s3-b001-f1",
      "severity": "S0",
      "title": "Workspace membership INSERT policy is self-scoped",
      "file": "db/migrations/2026…_workspace_members.sql",
      "line": 141,
      "claim": "Any authenticated user can insert a membership row for any workspace id they hold.",
      "failure_scenario": "Authenticated user U (no membership in workspace W) POSTs a row {workspace_id: W, user_id: U} over the REST layer. WITH CHECK is (user_id = auth.uid()) only. Row lands; U is now a member of W and, via the auto-join trigger at line 160, of W's organization.",
      "evidence": "L139-143: CREATE POLICY members_insert ON workspace_members FOR INSERT WITH CHECK (user_id = auth.uid());",
      "why_our_gates_missed_it": "The self-scoped-write lint keys on a table having an organization_id column; this table reaches its tenant through workspace_id only, so the lint never evaluated it.",
      "confidence": "certain",
      "suggested_fix": "WITH CHECK must also require the caller to be an owner or manager of workspace_id (or an org manager of its org). One ALTER POLICY."
    }
  ],
  "checked_found_none": [
    { "category": "SECURITY DEFINER exposure", "files": ["db/migrations/2026…_workspace_members.sql"], "note": "no definer functions in this batch" },
    { "category": "service-role client without re-applied scope", "files": ["src/app/api/workspaces/[id]/members/route.ts"], "note": "uses RLS client throughout" }
  ],
  "notes": [
    "The auto-join trigger amplifies any membership-insert bug into an org-membership bug; worth a seam finding for the chair."
  ]
}

Three parts of this shape do the work and should survive any redesign of your own tooling:

• ranges_read with skipped and skip_reason. This is what turns coverage into a fraction. In August, 780 of 781 units range-checked clean; the one that came back short was a batch of migration files whose seed middles the reader had sampled, and because the ledger caught it rather than hiding it we could look, reclassify the middles out of scope with a reason, and move on.
• why_our_gates_missed_it. A bug that survived your checks is also a statement about your checks, and that statement is usually worth more than the bug. It is the raw material for Stage 4.
• checked_found_none. An audit that reports only findings cannot be compared to the next one and cannot be refuted. Our August ledger carried 7,520 of these attestations beside 6,464 findings.

3.5 The orchestrator's rules

The orchestrator is a script, not a person. Its job is to be boring and unbribable.

ORCHESTRATOR RULES · STAGE 1

- A unit is `reviewed` when the union of ranges_read plus skipped covers
  [1, lines_total] and every skipped range has a reason. Otherwise it is
  `partial` and is re-dispatched with the gap named.
- A batch that returns no JSON, or JSON that fails the schema, is
  `failed`, not `reviewed`. It is retried once with the same brief, then
  surfaced to a human. An absence is never counted as a pass.
- The ledger (findings.jsonl, receipts.jsonl) is append-only and is the
  only coordination mechanism. Two sessions may work the same seat in
  parallel; they reconcile by unit id, never by conversation.
- Every long run is assumed to be killed halfway. Checkpoint after every
  batch; resume re-dispatches only units still reading `unreviewed` or
  `partial`; completed batches replay from the ledger, not from the model.
- Findings are de-duplicated by (file, line, claim-hash) at the end of
  the stage, not during it. Two seats filing the same defect from
  different starting questions is corroboration, and it is recorded as
  such: it is the strongest single signal in the corpus.

3.6 Harness lessons from Stage 1

The harness broke more often than the codebase did. Three things to build in from the start:

Schema caps fail silently. Our first structured-output schema put maxLength limits on finding fields. When a reviewer had more to say than the cap allowed, the response failed validation, blew the retry budget, and the batch returned nothing. Not an error we saw, just an absence. Nineteen batches had to be re-run against a relaxed schema. A constraint that turns a full finding into an empty result is a data-loss bug in a validation costume. Cap nothing; truncate at the consumer if you must.

Long runs die of inactivity. Background passes long enough to be useful hit session limits and drop. The fix is not a bigger timeout; it is resumability. Slice, checkpoint, re-dispatch only what still reads unreviewed.

The work outgrows one session. Our largest pass finished across two sessions running in parallel on one branch, with a shared id map reconciling the two finding streams. That worked only because the ledger was the coordination mechanism. Two sessions with prose reports and no ledger produce two irreconcilable opinions.

3.7 Stage 1 exit criteria

• Every review unit reads reviewed or ejected with a reason. Report the fraction; we hold ourselves to 99.5% or better.
• Every finding has a quoted evidence line, a concrete failure scenario, and a named gate that missed it.
• Every seat has filed checked_found_none for every category in its lens.
• The ledger is committed at the pinned commit, with the batch index beside it.

---

Part 4 — Stage 2: Refutation

Coverage gets you a pile. It says nothing about whether the pile is true. Stage 2 is the layer that makes the output trustworthy rather than voluminous, and it is the layer most teams skip because it feels like paying twice for the same reading. It is not the same reading. It is the opposite reading.

4.1 The three rules of refutation

Rule 1: a different model family. Two sessions of one model family share their blind spots exactly. "Independent" then means a different conversation, not a different mind. When our fan-out ran on Claude, refutation ran on the OpenAI family, and we reverse the pairing on the next pass. When we ran a single-family pilot early on, the refuter agreed with the finder at a rate that told us nothing.

Rule 2: no anchoring. The refuter receives the finding's claim, file, line and failure scenario. It does not receive the finder's evidence quote, the finder's reasoning, or the finder's suggested fix. Anchoring is how you pay for model diversity and receive an echo back. A brief that leaks the finder's hypothesis has spent the money and bought nothing.

Rule 3: disprove, not assess. The instruction is not "evaluate this finding". It is "prove this finding wrong". A refuter asked to evaluate will produce a balanced paragraph. A refuter asked to disprove will go and read the code at the actual commit, construct the counterexample, and either fail (the finding is confirmed) or succeed (the finding is dead). Both outcomes are useful; only the adversarial framing produces them.

4.2 The four attack vectors

Every refuter is required to try all four before returning a verdict:

1. The mechanism is wrong. The defect is real but the stated cause is not; the hole is one table over, or on the background path nobody mentioned, or in the sibling policy. This is the most valuable outcome: the finding is kept but re-aimed. Six of our confirmed August findings had the wrong stated mechanism. Hand any of those six to a developer as filed and you get an afternoon spent hardening a file that was never the problem, plus a closed ticket over a live defect.
2. Something upstream already neutralises it. A framework sanitiser, a middleware, a database constraint, a grant already revoked. Our best example: a stored-XSS claim via an href attribute. The refuter did not argue from memory about what React does with a javascript: URL. It downloaded the exact react-dom version, read the URL sanitiser in both renderers, rendered the case, and confirmed the framework emits its blocked-URL stub. Refuted, and while proving it, it found a real unrelated defect next door.
3. The failure scenario cannot actually be constructed. The role does not exist, the route is not reachable, the input is validated earlier, the data shape never occurs.
4. The proposed fix would break something. This one deserves a poster. A finding proposed adding an organisation filter to a query: plausible, idiomatic, matching the pattern used everywhere else. The refuter checked the data instead of the pattern. On the real schema, that filter matched zero rows. Shipping the prescribed fix would have silently turned off a working feature while closing a finding that was not real. A fix that turns a feature off and reports success is worse than the defect it resolved.

4.3 The refuter brief

BRIEF · STAGE 2 · REFUTER

You are an adversarial reviewer. A different model, in a different
session, has filed the finding below against the codebase pinned at
commit {PIN}, checked out read-only at {REPO_PATH}. You have not seen
that model's reasoning and you will not. Your only job is to PROVE THE
FINDING WRONG. If you cannot, say so with evidence, and say precisely
what survives.

THE FINDING
  id:               {id}
  severity filed:   {severity}
  file:             {file}
  line:             {line}
  claim:            {claim}
  failure scenario: {failure_scenario}

WHAT YOU MUST DO, IN ORDER
1. Open the file at the commit. Read the whole function or policy the
   line sits in, and every caller or dependent you can reach in the
   repository. Quote what you read.
2. Attack vector A — the mechanism. Is the stated cause the real cause?
   Trace the actual data flow. If the defect is real but the cause is
   elsewhere, say where, with a quote.
3. Attack vector B — upstream neutralisation. Search for anything that
   makes the failure scenario impossible before it reaches this line: a
   sanitiser, a validator, a middleware, a constraint, a grant. If it is a
   library behaviour, READ THE LIBRARY SOURCE at the pinned version; do
   not argue from memory.
4. Attack vector C — constructibility. Can the inputs in the failure
   scenario actually be produced by a realistic actor? Name the actor,
   the role, the route, the body. If a step is impossible, say which.
5. Attack vector D — the fix. If the finding implies a fix, ask what
   that fix would break. Check the schema, the data shape, the callers.
6. Only then, the verdict.

VERDICT VALUES
  confirmed   — the claim, the mechanism, and the scenario all hold;
                severity stays as filed
  downgraded  — real, but the blast radius is narrower than filed;
                give the new severity and why
  re-aimed    — real, but the mechanism or location is different;
                give the corrected file/line/claim
  refuted     — the failure scenario cannot occur; give the exact reason
  needs-live  — the verdict depends on production state you cannot see
                (a grant, a data shape, a config row); write the exact
                read-only query that would settle it

RULES
- Every claim you make cites a file and line at the commit and quotes it.
- You may not soften a verdict to avoid being wrong. "Probably fine" is
  not a verdict.
- If, while refuting, you find a DIFFERENT defect, file it as a new
  finding in `side_findings` with the full Stage 1 shape. Do not let it
  contaminate the verdict on the finding you were given.
- Do not propose designs. State the smallest change that would close what
  survives, in one sentence.

OUTPUT — JSON only:
{
  "id": "…", "verdict": "…", "severity_out": "S0|S1|S2|S3|null",
  "vectors": { "A": "…", "B": "…", "C": "…", "D": "…" },
  "evidence": ["file:line — quoted text", …],
  "corrected": { "file": "…", "line": 0, "claim": "…" } | null,
  "live_query": "…" | null,
  "smallest_fix": "…",
  "side_findings": [ … ]
}

4.4 Reconciliation

Send every S0 and S1 to at least two refuters. Send S2s to one if budget allows; we spot-check S3s and let them carry their filed severity into the hygiene backlog. Then reconcile mechanically:

RECONCILE RULES · STAGE 2

- A verdict survives only if EVERY refuter confirmed (or re-aimed to the
  same corrected location). One refutation kills it; the finder does not
  get a rebuttal round.
- Severity out = the MINIMUM across refuters.
- A re-aimed finding replaces the original's file/line/claim and keeps
  the original id with a `.r` suffix, so the audit trail shows the move.
- `needs-live` verdicts are collected into a single read-only query list
  and run by one engineer against production BEFORE Stage 3 starts.
  Nothing marked needs-live is ranked until its query has been run.
- Side findings enter the ledger as new Stage 1 findings and go through
  their own refutation in the next batch.

The needs-live list matters more than it looks. Some of our highest-severity findings turned on a single fact that the code could not settle: does the anonymous role actually hold EXECUTE on this function today? The read-only query is trivial:

select p.proname,
       has_function_privilege('anon',          p.oid, 'EXECUTE') as anon_can_exec,
       has_function_privilege('authenticated', p.oid, 'EXECUTE') as authed_can_exec
from pg_proc p join pg_namespace n on n.oid = p.pronamespace
where n.nspname = 'public' and p.prosecdef;

Run it, and half a page of hedging collapses into a yes or a no.

4.5 The second-path audit: an independent lane, not just a refuter

Refutation checks the findings your own family produced. It cannot find what your family never looked for. So in addition to refuting, we run the second model family as an independent auditor with its own lanes, briefed with the law and the intent but never with our hypotheses. This is the cheapest way to learn what your gates are blind to.

The lane brief is short. The important part is the standing preamble and the three schema fields it demands:

BRIEF · STAGE 2b · SECOND-PATH LANE (independent family)

You are a second model family reading a codebase pinned at {PIN}, checked
out read-only at {REPO_PATH} with no secrets and no database. Read the
repository's engineering rules file first ({RULES_FILE}); it is the law
this code was written under.

Your value is that you are NOT the model that wrote this code. Another
family already reviewed it. If your finding restates reasoning the author
would have produced, it is worth nothing. What is worth something: a
mechanism nobody here traced, a counterexample nobody constructed, a
gate that looks load-bearing and is not.

THE LANE
{one of: verification-and-tests / tenancy-and-authz / the-gates-themselves /
 product-behaviour}
{lane-specific "where to look" and "what I want to know" — see 4.6}

RULES
1. A claim is not evidence. Open the mechanism.
2. Every finding needs a concrete failure with inputs.
3. Name the gate that failed and the mechanical reason it did not fire.
4. Do not propose a change the rules file forbids; propose changing the
   rule, and say why, in the same breath.
5. Bound yourself: fill looked_and_found_nothing. If you ran out of
   budget before reaching something, say so in coverage.
6. Severity S0–S3 as defined; do not inflate.

OUTPUT — JSON only, matching:
{
  "lane": "…",
  "coverage": "what you actually read, and what you did NOT reach",
  "findings": [ { id, title, severity, file, line, claim, failure_scenario,
                  evidence, why_our_gates_missed_it, confidence,
                  suggested_fix } ],
  "looked_and_found_nothing": [ "area — why judged sound" ],
  "gate_critique": [ "rule/gate — the class of bug it does not actually stop" ]
}

Our first full run of this produced 25 findings. A first-family seat then verified each against the code and, where the claim turned on live state, against production with read-only SQL. Twenty-five of 25 were confirmed on mechanism; seven were downgraded on severity; several of the most severe were latent (the mechanism was live, but the data that would trigger it did not yet exist). The gate_critique field alone produced 36 entries, and Part 7 of this guide is largely built from them.

4.6 The four lanes

Give the second family four lanes, each with a "where to look" list and a "what I want to know" list. Here is the tenancy lane as an example; write the other three in the same shape.

LANE · TENANCY ISOLATION AND AUTHORIZATION

The question: can one tenant reach another tenant's data?

Where to look
- the schema snapshot: every CREATE POLICY, read as a set. Which tables
  have a write policy whose only predicate is the actor?
- the permissions module and the active-context resolver
- every route handler: how it derives the organization or project it is
  about to write to, and whether that derivation is attacker-controlled
- the service-role client and every file that imports it: it bypasses
  row security entirely
- the agent-facing tool surface: every mutation
- the documented list of deliberate policy exceptions: verify each one
  is actually the thing it claims to be

What I want to know
1. A concrete cross-tenant path: actor, role, route, body, and the row in
   another organization that changes. Trace it end to end.
2. Where a SECURITY DEFINER function is callable by the anonymous or
   authenticated role and does more than its name implies.
3. Where the organization is resolved from the USER rather than from the
   RESOURCE, and what an operator who spans organizations does there.
4. The gap between what row security enforces and what the route assumes
   it enforces. Those two are checked by different people at different
   times.

The rules file names several checks in this area. Treat them as
hypotheses about where the danger is, not as proof the danger is handled.
Check whether each actually closes its class, including via its allowlist.

The other three lanes: verification (do the tests and acceptance probes actually fail when the feature is broken, or do they pass vacuously?), the gates (for each automated check: what evades it?), and product behaviour (does the running system do what its own documentation says it does, on the paths a user takes?).

4.7 Stage 2 exit criteria

• Every S0 and S1 has at least two refuter verdicts and a reconciled outcome.
• Every needs-live query has been run and its answer recorded beside the finding.
• The second-path lanes have returned and their findings have been verified by the first family.
• The ledger now carries severity_filed, severity_out, verdict, and verified_by per finding.

---

Part 5 — Stage 3: Cross-examination and synthesis

Ten seats produce ten partial pictures. Stage 3 produces one, and it starts by attacking the ten.

5.1 The devil's advocate

Before anything is ranked, one session reads every seat's verdicts and argues the strongest good-faith case against the emerging consensus. It is never skipped, even when the consensus looks obvious. Especially then.

BRIEF · STAGE 3 · DEVIL'S ADVOCATE

You have the reconciled ledger from Stage 2 and the ten seat summaries.
Read all of them. Then:

1. Identify the consensus. In three sentences: what do the seats agree
   is the risk posture of this codebase, and which five findings do they
   collectively treat as most important?

2. Construct the strongest good-faith argument that the consensus is
   wrong. Not a strawman. The argument a sharp, experienced engineer who
   had just joined would make after reading the same ledger. Consider:
   - Is the ranking static? Does it privilege what LOOKS bad in source
     over what production telemetry (Stage 0) says is actually hot?
   - Is a "finding" actually the system working as designed? A ratchet
     holding at its baseline is not a defect. A measurement TODO is not a
     finding. Idempotent boilerplate is not duplication.
   - Have two seats filed the same defect twice under different names,
     inflating its apparent weight?
   - Is the severity ladder being applied consistently across seats, or
     does one seat's S1 equal another's S3?
   - What did NOBODY look at? Name the blind spots: the surfaces no
     seat's lens covered.

3. Attack the weakest confirmed finding. Re-open its cited file yourself.
   If it should be killed or downgraded, say so and why.

4. For each of the top five findings, state what evidence would settle
   whether it deserves its rank, and whether that evidence exists in the
   ledger or in the Stage 0 sheet.

5. If the seats' verdicts diverge on any item, identify the sharpest
   conflict and state it as a question the chair must answer.

You MUST argue against the consensus even if you privately agree with
it. You do not vote and you do not rank. You produce a contrarian brief.

OUTPUT — JSON:
{
  "consensus": "…",
  "strongest_objection": "…",
  "structural_objections": [ "…" ],
  "kills": [ { "id": "…", "reason": "…" } ],
  "downgrades": [ { "id": "…", "from": "S1", "to": "S2", "reason": "…" } ],
  "blind_spots": [ "…" ],
  "settling_evidence": [ { "id": "…", "evidence_needed": "…", "exists": true } ],
  "sharpest_conflict": "…"
}

In our August multi-role review the devil's advocate produced three kills, five downgrades, eight named blind spots (mobile, billing, non-English locales, data erasure, storage, the onboarding funnel, and the observation that nobody had run the app), and one structural objection that changed the program: the specialist phase had run static-only, so its ranking was a ranking of what looked bad in source. The chair's answer was to make Stage 0 measurement a gate, and it has been one since.

5.2 The seam benches

Some risks live between lenses, and no single seat owns them. A seam bench is a short session given two seats' outputs and one question. We ran four in August; the questions are reusable.

BRIEF · STAGE 3 · SEAM BENCH

You have the reconciled findings from seats {A} and {B}. Your question:

  {one of:
   · "Row-security posture versus database cost: where does the tenancy
      seat's proposed policy hardening collide with the performance
      seat's per-row re-evaluation findings, and what is the policy shape
      that satisfies both?"
   · "The agent loop versus the prompt: where does the agentic seat's
      autonomy finding depend on a prompt the prompt seat rated
      ungrounded, and which fix comes first?"
   · "Latency attribution: for each slow route in Stage 0, which seat's
      finding explains the number, and where does no finding explain it?"
   · "Cache versus freshness: where does the performance seat's caching
      proposal invalidate a correctness assumption the backend seat
      documented?"}

Produce:
1. The collision list: pairs of findings that cannot both be fixed as
   filed, with the reason.
2. The dependency list: findings whose fix must precede another's.
3. The gap list: numbers in Stage 0 that no finding explains.
4. One recommended resolution per collision, in one sentence each.
JSON only.

5.3 The chair

The chair is one session, briefed as the CTO's synthesis architect, and its output is the plan. Its first job is to answer the devil's advocate's structural objection before ranking anything. Its second is to route one hundred percent of confirmed findings somewhere. Not the top fifty. All of them, even if the destination is "appendix, hygiene, not this quarter".

BRIEF · STAGE 3 · CHAIR / SYNTHESIS

You are the review lead. Inputs: the reconciled ledger, the Stage 0
measurement sheet, the ten seat summaries, the devil's advocate brief,
and the seam bench outputs. Produce `audit/SYNTHESIS.md` with exactly
these sections.

1. RISK POSTURE HEADLINE (one paragraph)
   What is the ONE pattern behind the confirmed top-severity findings?
   If there is one, name it; it is the thing to fix structurally. If
   there is not, say so.

2. THE STRUCTURAL OBJECTION, ANSWERED
   Quote the devil's advocate's strongest objection and answer it. If
   the answer changes the program (a new gate, a new stage), write the
   rule in one sentence.

3. CROSS-EXAMINATION OUTCOMES
   Every kill, merge, downgrade and split from Stage 2 and the devil's
   advocate, as a table: id · outcome · one-line reason.

4. COVERAGE STATEMENT
   Units reviewed / total; findings filed; attestations filed; findings
   refuted by band; what was NOT covered and why. Numbers, not adjectives.

5. THE WAVE-SEQUENCED BACKLOG
   Wave 0 · live-verify: every needs-live query, half a day, read-only.
   Wave 1 · the confirmed S0s, cheapest and scariest first, one row each:
            id · fix · size (XS/S/M/L).
   Wave 2 · structural fixes that kill whole classes. Each row: the
            class · the findings it covers · the ONE change · the
            ENFORCEMENT (the check that makes it impossible to reintroduce).
   Wave 3 · remaining S1s by class, one PR per class.
   Wave 4 · re-refute anything still pending.
   Wave 5 · the S2/S3 tail, appendixed.
   Cap the actionable backlog at fifty rows. If more than fifty findings
   deserve action, the wave structure is wrong; merge by class.

6. THE ROUTING LEGEND
   Every confirmed finding id appears exactly once with a destination:
     W#   — in wave #
     INV  — becomes an invariant check (name the rule)
     ADR  — needs an architecture decision record first
     HUMAN — needs a decision only the founder/CEO can make (state it)
     AC   — accepted as-is, with the reason
     W0   — blocked on a live-verify query
   Unrouted findings are a defect in this document.

7. LEAVE-ALONE VERDICTS
   Findings the seats filed that you have decided NOT to act on, with
   the reason. This section is as important as the backlog.

8. DECISIONS FOR THE CEO
   At most five. Each: the decision, the two options, your recommendation,
   what it costs to be wrong.

9. THE ONE ENFORCEMENT THAT MATTERS MOST
   If only one thing ships from this review beyond the S0 fixes, what
   is it, and why would it have caught every confirmed S0?

10. DONE MEANS
   A date, and the two numbers that must have moved by then: coverage
   re-run clean at the new commit, and the confirmed-finding count.

The "one enforcement that matters most" section forces the chair to think about class rather than instance. Ours, in August, was a hermetic row-security test harness: a database container seeded from the schema snapshot, in which for every tenant table a non-member's token is proven unable to read or write another tenant's row. Every confirmed critical in that review would have been caught by it, and unlike a grep-based check it verifies the live policy behaviour, not the code's intent. The review was declared done when that harness was green, not when the synthesis document existed.

5.4 Stage 3 exit criteria

• The devil's advocate brief exists and the chair has answered its structural objection in writing.
• Every confirmed finding appears exactly once in the routing legend.
• The actionable backlog has at most fifty rows.
• The "done means" section names a date and two numbers.

---

Part 6 — Stage 4: Fix and promote

A review is not finished when the report lands. A report is an inventory of intentions. Stage 4 is where the intentions become code, and, more importantly, where they become checks.

6.1 The quick-win gate

Some fixes ship during the review itself. The rule for which ones:

QUICK-WIN GATE
A fix may ship during the audit, ahead of the wave plan, only if ALL hold:
  - the finding is confirmed (survived Stage 2)
  - the fix is XS or S
  - it touches no migration, no authentication path, and no production
    data write
  - the change is under ~50 lines and isolated to application code
  - a test covers it, or the reproduction was run before and after
Everything else waits for its wave.

Our June pass shipped seven pull requests during the review under this gate, including the only critical. Our August pass held every critical for Wave 1 because most of them were policy changes, and policy changes wait for a migration slot.

6.2 The promotion brief

For every Wave 2 row, and for every confirmed finding whose why_our_gates_missed_it names a mechanical reason, one session writes the check. The brief:

BRIEF · STAGE 4 · PROMOTE A FINDING CLASS INTO A CHECK

Finding class: {one-line description}
Instances in this audit: {ids}
Why existing gates missed it: {from the ledger}

Write a check that runs in CI on every pull request and fails when a NEW
instance of this class is introduced. Requirements:

1. The check reads the repository, the schema snapshot, or a derived
   artifact. It does not need a running app or a database unless the
   class genuinely cannot be detected statically; if so, say why and
   propose the hermetic fixture it needs.

2. If the codebase already contains instances that cannot all be fixed
   now, the check is a RATCHET: it records today's count per file in a
   committed baseline file, and fails on any GROWTH. The baseline may
   only shrink. Regenerating the baseline is a deliberate command, never
   a side effect of running the check.

3. The check compares against the MERGE BASE, never against the working
   tree. A baseline compared to itself can be raised in the same PR that
   introduces the regression.

4. The check must fail when it cannot parse its input. A policy with a
   name it cannot tokenise, a file it cannot read, a JSON it cannot
   decode: each is a FAILURE naming the input, never a skip.

5. Provide an escape hatch that leaves a trace: a comment token with a
   mandatory reason (e.g. `// <rule>-ok: <reason>`). A bare token with
   no reason fails.

6. Prove the floor. Delete one real instance from the baseline, run the
   check, and paste the failure output. A check that passes with an
   emptied baseline is a check that never ran.

7. Add one row to the rules file: rule id · enforced by · one-line
   description · the incident it was born from.

Output: the check (shell or script), the baseline file, the regeneration
command, the proof-of-floor output, and the rules-file row.

6.3 The ratchet shape

Most of our checks are grep-based ratchets, and the shape is simple enough to paste. Here is a representative one, in the shell form our CI runs:

section "R26: bare Supabase write chains discard { error }"
# supabase-js never throws; a statement-position await on a write chain
# silently loses the failure. Ratchet: no growth vs the committed baseline.
BASELINE=scripts/r26-bare-write-baseline.txt
current=$(grep -rnE '^\s*await\s+\w+\.from\([^)]*\)\.(insert|update|delete|upsert)\(' src \
          --include='*.ts' --include='*.tsx' \
          | grep -v 'write-failure-ok:' \
          | cut -d: -f1 | sort | uniq -c | awk '{print $2"\t"$1}')
if [ "$WRITE_R26_BASELINE" = 1 ]; then printf '%s\n' "$current" > "$BASELINE"; fi
# Compare per file against the merge base's baseline, not the working tree's.
base=$(git show "$MERGE_BASE:$BASELINE" 2>/dev/null || cat "$BASELINE")
growth=$(join -t $'\t' -a1 <(printf '%s\n' "$current" | sort) <(printf '%s\n' "$base" | sort) \
         | awk -F'\t' '{ if (($3+0) < ($2+0)) print $1": "$3" -> "$2 }')
if [ -n "$growth" ]; then
  echo "R26 FAIL — bare write chains grew:"; echo "$growth"
  echo "Destructure { error } and surface it, or annotate '// write-failure-ok: <reason>'."
  fail=1
fi

The details are yours to adapt. The properties are not negotiable: a committed baseline, per-file counts, comparison against the merge base, a reasoned escape token, and a regeneration command that a human runs on purpose.

6.4 What gets promoted

Not every finding becomes a rule. The criteria:

1. It is a class, not a one-off. Two instances in the ledger, or one instance plus a plausible mechanism for recurrence.
2. It is mechanically detectable with a low false-positive rate. If the check needs a human to decide, it is a review convention, not an invariant, and it goes in the rules file as a convention with a reviewer prompt instead.
3. The incident is written down. Every rule row names the finding or incident it was born from, so a future engineer who wants to delete it knows what they are re-opening.

A sample of the classes that earned a check in our program, described at class level:

• A privileged database client used anywhere without an explicit allowlist entry and a written justification; the allowlist may only shrink.
• A scheduled job authenticating with an inline secret comparison instead of the shared helper (the inline version accepted an unset secret).
• A write policy on a tenant table whose only non-admin predicate is the actor's own id.
• A SECURITY DEFINER function newly executable by the anonymous or authenticated role; the exposure set is frozen and may only shrink.
• A read of an organisation-scoped table filtered on the user id alone, in a product where operators span organisations.
• A background job enqueue whose returned error is discarded.
• A fan-in over settled promises that filters to fulfilled values and never counts the rejections.
• A fetch inside an effect with no abort signal.
• A file over a size limit outside a frozen baseline.
• An LLM call site whose system prompt is an inline string rather than a resolved, versioned registry entry.
• An LLM call site with no recognised grounding helper.
• A generator with no declared feedback seam.
• A permission check that does not state its posture toward cross-organisation operators.
• A migration containing DROP or RENAME with no rollback comment.
• A prompt slot with the default character cap rather than a named one (the default silently truncated a document to a fifth of its length).
• A secret passed on a command line in CI (world-readable in the process table on shared hosts).

We have about a hundred and twenty such rules now, thirty-nine of them ratchets with baselines. The list grew by roughly one rule per confirmed class per audit, and the rules file's summary table is the closest thing we have to a written history of what nearly went wrong.

6.5 Stage 4 exit criteria

• Every Wave 1 row is merged and content-verified on the default branch, not merely on a feature branch.
• Every Wave 2 row has shipped its enforcement, with the proof-of-floor output in the pull request.
• Coverage has been re-run at the new commit and the confirmed-finding count has dropped to the number the chair named.
• The rules file has one new row per promoted class, each naming its origin.

---

Part 7 — Audit the machine

This is the part we got wrong first and the part most guides omit. The gates are software. The audit harness is software. Both were written quickly, mostly by the same agents, and both fail in exactly the way that is hardest to notice: they pass.

7.1 The seven ways a gate lies

From the gate_critique fields of our second-path audit and from our own incidents, grouped by mechanism:

1. A gate that cannot parse its input passes it. Our self-scoped-write lint silently skipped 134 policies whose names contained spaces, because the tokeniser returned nothing and the loop said continue. Two of the most severe findings in the audit were among the skipped policies. Rule: an unparsed input fails by name.

2. A gate keyed on a column name misses indirect tenancy. The same lint looked for an organisation or project column on the table itself. A table that reaches its tenant through a foreign key to a foreign key was invisible by construction. Rule: derive the tenant graph from the schema's foreign keys, not from column names.

3. A generator and its check share one bug. Our rules file is generated into a second-agent-family's rules file by a script, and a check verifies the two match. Both used the same cell-splitter, so a row truncated at an escaped pipe was "regeneration-clean". Rule: a derived-artifact check only proves agreement with its generator; test the generator separately against a hand-written expectation.

4. Ratchets trusted the working tree. Every per-file baseline was compared to the baseline in the PR, so regenerating it in the same PR raised the floor. When we fixed this we found sixty-one unearned credits across seven ratchets. Rule: compare against the merge base.

5. Grep gates read text, not programs. A guard named in a comment counted as a guard. A filter inside a comment counted as a filter. A file-wide exemption token exempted lines it was never meant to cover. Rule: strip comments before matching, scope exemptions to a line, and, where the class is important enough, parse the AST.

6. Required gates that skip where they matter. Our row-security coverage check had no database to test against on feature branches, so it self-skipped, and it was required. A skip reads as green. Rule: a required check that cannot run must fail, not skip; and for every required check, name the program that writes the data it reads and prove that program runs on that lane.

7. No gate at all for authorisation across arguments. Three of the most severe second-path findings had the same shape: authorise against one caller-supplied id, act on another. Nothing in our rule set asked "do these two ids belong to the same tenant?" Rule: when the finder names a class no gate covers, that is a Wave 2 row, not a fix.

7.2 Vacuous passes

A probe that skips reads as green. So does a count satisfiable by absence. When we audited our own end-to-end acceptance suite, five of fifteen specs could pass with the feature they guarded completely removed: a spec that asserted "no error banner" passed on a blank page; a spec that counted rows passed on zero rows; a spec that skipped when a fixture field was missing skipped every run because the field had been renamed.

BRIEF · STAGE 2b · LANE: VERIFICATION

The question: do the tests and acceptance probes in this repository
actually fail when the thing they guard is broken?

For every end-to-end or acceptance spec:
1. State in one sentence what user-visible outcome it claims to guard.
2. Construct the cheapest breakage of that outcome (delete the button,
   return an empty list, throw in the handler).
3. Say whether the spec would fail, pass, or skip under that breakage,
   with the assertion line quoted.
4. Any spec that passes or skips under breakage is a finding at S1: the
   feature is unguarded while appearing guarded.

For every `skip` in the suite: what condition triggers it, how often is
that condition true in CI today, and what does the skip hide.

For every assertion of the form "count >= N" or "no X present": can it
be satisfied by the feature being absent entirely?

7.3 False floors

A ratchet's floor is only real if deleting a real instance from the baseline makes the check fail. We learned this when a type-escape ratchet reported one file with eight sites, and a hand count found 174 files with 452 sites: the pattern had a bug, the baseline froze the bug's output, and the check had been green for weeks because nothing could grow past a floor that low. The proof-of-floor step in the promotion brief exists because of this. Do it for every ratchet you already have, not just new ones:

for b in scripts/r*-baseline.txt; do
  cp "$b" /tmp/bak; sed -i '1d' "$b"           # remove one real entry
  if bash scripts/check-invariants.sh >/dev/null 2>&1; then
    echo "FALSE FLOOR: $b — check passed with an entry removed"
  fi
  cp /tmp/bak "$b"
done

7.4 The harness reads as coverage

Our nightly QA run reported 248 failures on one night. Seventeen were real. The rest were the harness: 29 copy-pasted spec files that shared one broken fixture, a watchdog that killed the type checker mid-run and printed zero errors (a killed type check looks clean; only the exit code says otherwise), and a login step that timed out and cascaded. A harness that fails loudly on its own bugs is a harness that gets fixed. A harness that fails quietly trains everyone to ignore red.

Two rules from that night. First, a killed or timed-out check is a failure with a distinct signature, never a pass and never a generic failure. Second, no two spec files may share a fixture by copy-paste; the twenty-nine became one.

7.5 The autofix loop is a gate too

If you run an autonomous fix loop (an agent that triages bugs and opens pull requests), it is part of the machine and it needs the same scrutiny. Our worst incident in this area had no code defect at all. A new required check landed without a writer on the branch it ran on, and the merge rate went from eighty-one merges a day to forty, then to one, over four days, while every dashboard was green because the check was "passing" by skipping. The lesson is in rule 6 above and is worth its own sentence: for every required gate, name the program that writes the data it reads, and prove that program runs on that lane.

7.6 Acceptance escapes

The last class of failure is the one no automated gate catches: the code did what the code intended, and nobody verified it did what was asked. We keep a ledger of these. Every row is a shipped change that passed every check and did not deliver the outcome the requester wanted, discovered by a human meeting the feature in the running app. Sixteen rows in ten weeks. Reading them is humbling and useful: the fix was verified on fresh state while the reporting browser carried old state; the fix covered new records and not the one on the requester's screen; the fix was verified on a single-item fixture and the count came from multi-item data.

The countermeasure is procedural. Before building anything user-facing, the builder writes one sentence in the form given this state, when this action, then this visible result, describing what was asked, not what will be implemented. After building, the pull request carries a screenshot from the running app proving that sentence. Green checks are not that proof. They verify the code does what the code intends, which is the definition of the failure mode.

ACCEPTANCE ESCAPE LEDGER · row shape
| # | Reported | Escaped in | What was asked | What shipped instead | Caught by | Fixed in |
Append a row whenever the requester says "that's not what I meant".
Bugs (code not doing what its author intended) do NOT go here.
If rows keep landing after the outcome-sentence + screenshot rule, the
process is still broken; iterate on the process, not the row.

---

Part 8 — Governance that makes it stick

An audit that is not wired into the delivery process decays in a quarter. Three pieces of governance hold ours.

8.1 Required checks and a merge queue

Every promoted rule runs in one CI job. That job is required. Nine checks are required on our default development branch: type check, the invariants job, lint at zero warnings, row-security coverage, unit tests, an authorisation matrix, translation key parity, a build with a function-size gate, and the change-band gate described next. Everything else (performance budgets, schema drift, a browser end-to-end pass) is advisory: it reports, it never blocks.

The split matters. A required check that is flaky trains people to override it. An advisory check that is important gets promoted once it has run reliably for two weeks. And the merge queue, not a human, lands green pull requests, so the incentive to "just merge it" never meets an opportunity.

8.2 Change authority follows blast radius, not the person

We classify every pull request into a band by the files it touches, mechanically:

• direct: the blast radius stops at the build. Anyone may land it on green.
• owner: one domain's owner accepts it. Named in the pull request.
• founder: the application does something different for every user, or the change cannot be cleanly undone. The founder accepts it. Migrations, permission code, billing, anything that writes production data.

The classifier can only ratchet a band up. Declaring founder on a documentation change is fine. Declaring direct on a diff that touches the permissions module is the one thing that fails. This exists because prose in a pull request body is not a review surface; a label the search can find is.

One related lesson: a stacked pull request defeats band review. Merging the child lands the parent's code, so a founder gate on the parent is bypassed silently. Do not stack across bands.

8.3 Cadence and triggers

• Quarterly: the full five-stage program.
• Continuous: the promoted checks, on every pull request.
• Out of cycle, any of: a production incident that traces to a bug class; a migration touching five or more tables; a bundle size jump over twenty percent; adoption of a new model tier (every prompt gets a tier-scoped sweep); a new agent or tool surface with write access.
• After every pass: a "done means" date and two numbers. Coverage re-runs clean at the new commit; the confirmed-finding count has dropped to the target. The date goes in the synthesis so nobody can quietly let it slide.

---

Part 9 — Execution

9.1 The six-week calendar

Compress to three weeks for a codebase under a quarter of a million lines. Do not compress Stage 2. It is the stage that makes the other four worth doing.

9.2 The CTO's checklist

Print this. Tick it.

Before the audit

• A commit is pinned and every artifact will name it
• A detached checkout exists with no .env file and no credential in it
• The denominator is from git ls-files; ejections carry reasons
• The entity-class inventory is taken and compared to last pass
• The measurement sheet exists, every number with a query beside it
• LLM spend in the provider console reconciles with the call log within 10%
• Two model families are available; the pairing (finder / refuter) is decided and will reverse next pass

During Stage 1

• Each seat has one lens and a named practitioner in its brief
• Batches are three units; the index is committed
• Receipts carry line ranges; the orchestrator unions and re-dispatches gaps
• Output schema has no length caps
• Every run is resumable from the ledger
• Every seat files "checked, found none" per category

During Stage 2

• Refuters see claim, file, line, scenario. Not evidence, not reasoning, not fix
• Every S0/S1 has two refuters; survival requires unanimous confirmation; severity is the minimum
• Every needs-live query has been run before ranking
• The second-path lanes ran with the law and the intent, never with hypotheses
• Every gate_critique entry has been read by a human

During Stage 3

• The devil's advocate ran even though consensus looked obvious
• The chair answered the structural objection in writing before ranking
• Every confirmed finding appears once in the routing legend
• The backlog has at most fifty rows
• "Done means" has a date and two numbers

During Stage 4

• Quick wins met all five gate conditions
• Every class fix shipped its check; each check compares to the merge base; each has a reasoned escape token
• Proof of floor pasted for every new and every existing ratchet
• Rules file has one row per promoted class, each naming its origin

Auditing the machine

• Every required check has a named writer for the data it reads, proven to run on that lane
• A check that cannot parse its input fails by name
• Killed or timed-out checks have a distinct failure signature
• Acceptance specs have been tried against the cheapest breakage; vacuous ones fixed
• The acceptance-escape ledger exists and the outcome-sentence + screenshot rule is in the PR template

Governance

• Required vs advisory checks are split; the queue lands green PRs, not a human
• Change bands are classified from file paths and can only ratchet up
• The next quarterly pass is on the calendar with its owner

9.3 Five questions for the CEO to ask when it is done

1. "What fraction of the code was read, and how do you know?" The good answer is a number over a denominator from git, with the ejected files and their reasons. "All of it" without a number is the bad answer.
2. "How many findings did the refuters kill or downgrade?" The good answer is somewhere between a third and two thirds of the top band. Near zero means the refuter was anchored or came from the same family. Near all means the seats were sampling.
3. "What is the one pattern behind the criticals, and what check now makes it impossible?" The good answer names a class and a CI rule with a proof-of-floor. A list of fixed tickets with no rule is the bad answer.
4. "Which of our existing checks turned out to be lying?" The good answer is a short list with mechanisms (could not parse, compared to itself, skipped where required). "None" is almost certainly wrong.
5. "What is the date, and what are the two numbers?" Coverage re-run clean at the new commit, and the confirmed count at target. If there is no date, the audit is a document, not a program.

9.4 Start with one wave

If six weeks is more than you can commit to this quarter, run Stage 0 and one seat. Pick the tenancy lens if you are multi-tenant, the security lens if you are not. Route every route handler and every policy to it. Send every S0 and S1 it files to a refuter from a different model family, with the no-anchoring rule. Read what survives.

You will learn two things in a week: whether your codebase has the shape we found (a hardened path exists, and a later path skips it), and whether your existing checks would have caught it. Both answers tell you what the next five weeks should be.

---

Index of briefs in this guide

Everything in this guide was learned by getting it wrong first. The denominator exists because a report once claimed full coverage with no way to check. The refuter exists because a first-read severity is a hypothesis and half of hypotheses are wrong. The proof-of-floor exists because a ratchet held green for weeks at a floor that was a bug. The escape ledger exists because green checks verified the code did what the code intended, sixteen times, while the person who asked for it was looking at a screen where it had not happened.

Make the denominator come from git. Hire someone whose job is to kill your findings. Promote every class into a check, and then audit the checks. Everything else is reading, and reading was never the hard part.