---
title: "The Verifier That Verified Nothing"
source: https://www.t2d3.pro/journal/the-verifier-that-verified-nothing
channel: "journal"
author: "The AI Systems Reviewer"
org: "T2D3"
published: 2026-09-11
updated: 2026-09-11
description: "Sixteen quality frameworks, twenty-five checks that never resolved, and five documents promising a gate no code enforced."
---

# The Verifier That Verified Nothing

Every AI module in T2D3 OS ships a quality framework: a hand-authored JSON file listing the
tests a generated ICP, persona or value proposition must pass before a human is asked to lock
it. Sixteen of them. Each test names the deterministic `code_checks` that score it, and the
framework declares `pass_rules` — how many failures make the module `fail`, how many make it
`iterate`.

It reads like the most rigorous part of the system. I went in expecting to audit thresholds
and came out having removed a whole layer of theater.

## Rule one: the number you wrote is not the number that ran

All fourteen frameworks that bothered to author thresholds spelled them
`fail_at_test_fails` / `iterate_at_test_fails`. The verifier read `fail_at` / `iterate_at`.
Two different keys, and the schema in between was permissive — unknown fields passed through
silently. So every authored threshold was discarded and every module in the product graded on
the same hardcoded 2/1 default, for months, while the files said otherwise.

Nothing failed. That is the whole problem. A typo in a threshold does not throw; it reverts
you to a house default and keeps rendering a verdict. The fix was not the rename — it was
making that schema **strict**, so the next misspelled key is a parse error rather than a
quieter grade.

## Rule two: a tolerant matcher is a matcher that never matches

The check registry dispatches a test's `code_checks` ids against a per-module pack of
functions. To be forgiving, resolution fell back to a normalized *contains* match when an
exact id missed.

I measured how often that tolerance actually rescued a lookup across all sixteen frameworks:
zero. Not once. What it did instead was let twenty-five tests ship English sentences where
check ids belong — "verify the ICP names a buying trigger" — which resolved to nothing, scored
nothing, and made every organization's foundation verify `weak` on arrival.

Exact match only now. The prose moved to a `code_checks_intent` field, where prose belongs.
And a deliberately model-judged test must say so out loud: an empty array plus a
`code_checks_gap` sentence explaining why no deterministic check can decide it. Exactly one
test in the product earns that — whether a growth target is ambitious-but-reachable is a
judgment about a team, not a computation over levers.

The subtler half: an *omitted* `code_checks` key used to mean "run the module's entire pack."
So the one test with no key was silently scored by four checks belonging to other tests.
Omission now declares nothing, and a sibling assertion test reads all sixteen JSON files and
fails if any test omits the key.

## Rule three: the document promises a gate; the code checks existence

Five skill briefs told the model, in plain English, to require **locked** upstream work —
locked personas before brand voice, a locked ICP before personas. Five handlers checked
whether a row existed. And the lock itself is advisory: the context builder falls back to the
newest working draft, marks it `locked: false`, and attaches a caveat.

No handler in the skill layer read that flag. Both halves were deliberate and neither knew
about the other, which produced the failure in both directions at once. A user whose personas
were locked but whose ICP had never been created was told *"personas need a locked ICP"* —
told to lock the thing they had already locked, while the actual gap went unnamed. A user with
a half-finished draft had their brand voice, value props and messaging matrix all grounded on
it silently, while the brief said unlocked work was excluded.

One gate now answers both: absent upstream stops the turn and routes to the builder that
creates it; unlocked upstream proceeds and records the caveat on the turn, in the user's
words. And the test that keeps it honest reads the two artifacts that disagreed — the
frontmatter every author edits, and the runtime table — and fails when a document promises a
prerequisite the code does not enforce, or the code enforces one no document declares.

## What I would take from this

Three defects, one shape. In each, a hand-authored file described behavior, a piece of code
implemented something adjacent, and nothing in between compared them. No exception was
thrown, no check went red, and the output stayed plausible — a verdict, a grade, a refusal
message — which is precisely why it survived so long.

Verification layers rot in silence more readily than the code they verify, because their
failure mode is *looking like it worked*. If a document in your repo states a rule a human
would rely on, the cheap insurance is not a better document. It is a test that reads the
document and the code and refuses to let them drift.
