The Gate Graded a Shape Nobody Wrote

The last pass through this codebase closed a gap that had been open since the quality frameworks were written: sixteen modules shipped a JSON file describing the tests their output must survive, and two of them shipped it to nobody. Nothing called the verifier. The fix was a registry — every module type declares whether it is gated, by which handler, or says in prose why it is not — and a test that fails CI when the declaration stops being true.

That closed the question "does anything verify this?". It did not close the question underneath it, which turns out to be the more interesting one: verifies it against what?

Wiring a gate is not the same as converging a shape

customer_interviews was one of the two ungated frameworks. Turning the gate on meant finding the path that produces the artifact and calling verifyFramework there. Done, and the registry test proves the call is still in the file.

Except the framework's check pack — the deterministic half, the code that actually reads the generated rows and scores them — was reading module_items rows of type interview, each carrying a segments array of diarized speaker turns. That was a real shape once. It stopped being written months earlier, when customer-interview-coach was demoted from a builder to a router and the module handler took over the filing, writing interview_response and interview_script rows instead.

So the gate was on, the test was green, the registry said "gated", and the pack scored zero rows every time — because the rows it looked for did not exist. A verifier that finds nothing does not fail loudly. It reports what it found, which is nothing, which grades as absent evidence, which is indistinguishable from a genuinely thin interview.

The convergence is the unglamorous half of the work: find every writer of the module's rows, make them all emit one shape, wire the check ids against that shape, and put a source assertion beside the writers so the next person who adds a fourth writer with a fifth shape gets a red test instead of a quiet zero. We had done exactly this for personas one wave earlier — three writers, one still emitting a legacy nested row — and had learned nothing general from it, because it looked like a one-off.

Two one-offs is a class.

Which half of a per-module test generalizes

The honest question at that point is whether the per-module shape tests are a pattern that wants to be a single generic assertion. It is tempting to answer yes: both tests encode the same sentence — the pack reads what the writers write — and a generic version would cover the fourteen modules nobody has audited yet.

Pulling on it produced a three-way answer rather than a yes or a no.

"The pack reads a type somebody writes" is generic. Every handler-gated module has a pack; every pack's item_type === "..." comparisons are enumerable; the set of item types the product emits anywhere is enumerable too. A pack reading a type that appears nowhere in the emitted set is dead code by construction, and the assertion needs no per-module knowledge.

"The gate hands the pack something it can read" is generic. The registry already names each module's gating handler. The intersection of what that handler writes and what the pack reads must be non-empty, or the gate is decorative. Also enumerable, also per-module-blind.

"Every writer emits the converged shape" is not generic, and this is the part worth saying out loud, because the instinct to unify is strong and it would have produced a worse test. That assertion needs the writer set, and the writer set is genuinely per-module knowledge: personas has three writers scattered across a skills handler, a persistence helper and a survey apply-path; customer_interviews has exactly one choke point that both its ingestion paths funnel through. Nothing in the type system, the registry, or the file layout tells you which is which. A generic version would have to guess, and a test that guesses its own inputs fails for the wrong reasons or passes for no reason.

So: the two generic assertions landed as one new file covering every gated module. The two per-module shape tests stayed exactly where they were. The finding is not "we should have generalized sooner" — it is that a per-module test can be half-generic, and the useful move is to lift the half that is, not to hold the whole thing hostage to the half that is not.

What the generic pass found, and what we did not fix

Building the sweep produced one immediate false positive worth recording, because it is the kind of thing that makes people abandon generic tests. The naive diff — types the pack reads, minus types the handlers write as string literals — flagged personas as reading three types nobody wrote. It writes them via item_type: tier, a variable in a loop. There is no literal to grep.

The fix was not a special case. It was noticing that the module type catalog already declares every item type each module is allowed to produce, per module, by hand — which is precisely where a dynamically-emitted type is written down. Unioning the catalog into the emitted set removed the false positive without weakening the assertion, and the negative test still fails loudly when a real type is misspelled.

Two genuine findings survived, and neither is fixed:

The messaging_framework pack reads a messaging_cell type the module no longer writes — it writes message_cell, a deliberately distinct artifact. That read is dead for anything generated today. It is not dead for rows written before the rename, which still sit in the production database. So it stays, as an allowlist entry with a reason longer than eighty characters, in a list the test allows to shrink and never to grow. An allowlist that costs a paragraph to add to is an allowlist people empty.

The icp pack requires economics metrics — CAC payback plus one of NRR or LTV:CAC — that the ICP builder deliberately never drafts, because they are the customer's real numbers and inventing them would be worse than leaving them blank. The builder even says so in its completion summary: fill it in the Economics tab. Which means the check is a guaranteed failure on every freshly built ICP, and it is one of the two failures that flip the module's verdict from needs_iteration to fail.

The obvious fix — mark it not-applicable when the module was skill-built — would also silence it for an ICP whose Economics tab is genuinely empty months later, which is the case the check exists to catch. That is a product decision about what a fresh ICP's verdict should say, not a test cleanup, so it is written down where the next pass will find it rather than quietly patched into passing.

Both of those are the same discipline: the value of a sweep is the findings it surfaces, and a finding you paper over to get a green run is a finding you paid for and threw away.