The Verifier That Verified Nothing

Every AI module in T2D3 OS ships a quality framework: a hand-authored JSON file listing the tests a generated ICP, persona or value proposition must pass before a human is asked to lock it. Sixteen of them. Each test names the deterministic code_checks that score it, and the framework declares pass_rules — how many failures make the module fail, how many make it iterate.

It reads like the most rigorous part of the system. I went in expecting to audit thresholds and came out having removed a whole layer of theater.

Rule one: the number you wrote is not the number that ran

All fourteen frameworks that bothered to author thresholds spelled them fail_at_test_fails / iterate_at_test_fails. The verifier read fail_at / iterate_at. Two different keys, and the schema in between was permissive — unknown fields passed through silently. So every authored threshold was discarded and every module in the product graded on the same hardcoded 2/1 default, for months, while the files said otherwise.

Nothing failed. That is the whole problem. A typo in a threshold does not throw; it reverts you to a house default and keeps rendering a verdict. The fix was not the rename — it was making that schema strict, so the next misspelled key is a parse error rather than a quieter grade.

Rule two: a tolerant matcher is a matcher that never matches

The check registry dispatches a test's code_checks ids against a per-module pack of functions. To be forgiving, resolution fell back to a normalized contains match when an exact id missed.

I measured how often that tolerance actually rescued a lookup across all sixteen frameworks: zero. Not once. What it did instead was let twenty-five tests ship English sentences where check ids belong — "verify the ICP names a buying trigger" — which resolved to nothing, scored nothing, and made every organization's foundation verify weak on arrival.

Exact match only now. The prose moved to a code_checks_intent field, where prose belongs. And a deliberately model-judged test must say so out loud: an empty array plus a code_checks_gap sentence explaining why no deterministic check can decide it. Exactly one test in the product earns that — whether a growth target is ambitious-but-reachable is a judgment about a team, not a computation over levers.

The subtler half: an omitted code_checks key used to mean "run the module's entire pack." So the one test with no key was silently scored by four checks belonging to other tests. Omission now declares nothing, and a sibling assertion test reads all sixteen JSON files and fails if any test omits the key.

Rule three: the document promises a gate; the code checks existence

Five skill briefs told the model, in plain English, to require locked upstream work — locked personas before brand voice, a locked ICP before personas. Five handlers checked whether a row existed. And the lock itself is advisory: the context builder falls back to the newest working draft, marks it locked: false, and attaches a caveat.

No handler in the skill layer read that flag. Both halves were deliberate and neither knew about the other, which produced the failure in both directions at once. A user whose personas were locked but whose ICP had never been created was told "personas need a locked ICP" — told to lock the thing they had already locked, while the actual gap went unnamed. A user with a half-finished draft had their brand voice, value props and messaging matrix all grounded on it silently, while the brief said unlocked work was excluded.

One gate now answers both: absent upstream stops the turn and routes to the builder that creates it; unlocked upstream proceeds and records the caveat on the turn, in the user's words. And the test that keeps it honest reads the two artifacts that disagreed — the frontmatter every author edits, and the runtime table — and fails when a document promises a prerequisite the code does not enforce, or the code enforces one no document declares.

What I would take from this

Three defects, one shape. In each, a hand-authored file described behavior, a piece of code implemented something adjacent, and nothing in between compared them. No exception was thrown, no check went red, and the output stayed plausible — a verdict, a grade, a refusal message — which is precisely why it survived so long.

Verification layers rot in silence more readily than the code they verify, because their failure mode is looking like it worked. If a document in your repo states a rule a human would rely on, the cheap insurance is not a better document. It is a test that reads the document and the code and refuses to let them drift.