We Hired Three More AIs. None of Them Can Merge.

There's a version of "multi-model AI strategy" that's really just a dropdown: pick GPT or Claude or Gemini, hope for the best, switch when Twitter says something new is good. We wanted the opposite — a bench with defined roles, hard budgets, and a rule that keeps the whole thing honest:

Advisors advise. They never decide.

This week T2D3 OS's build system went from one primary AI (Claude, which writes most of the code) plus one second opinion (OpenAI's Codex, reviewing risky changes) to a genuine multi-vendor bench: Grok 4.6 joined the code-review panel, Gemini started doing nightly visual inspections of the product, and a nightly defect hunter now reads every merged change of the day. Here's the how, and then the two moments that made it worth writing about.

How: four rules before any vendor gets a seat

1. Everything cross-vendor is advisory. Our CI gates — the type checks, the invariant suite, the tests — remain the only thing that can block a merge. A second-opinion model can say HOLD, two high-severity findings; a human (or the primary AI, with the human's standing direction) decides what to do about it. No external model approves, merges, or blocks. This isn't ceremony — it's what lets us accept high-coverage, medium-precision reviewers without handing them the keys.

2. Nobody touches production traffic without winning an eval. If we want to move a live AI task from Claude to a cheaper model, the candidate has to beat the incumbent in a blind, position-swapped judgment over real production inputs — not a benchmark, our actual workload. No pass, no traffic.

3. Every vendor gets a hard, fail-loud budget cap. Grok has $25/month. Gemini's visual pilot has $10. The caps fail closed: if the spend ledger can't be read right before a paid call, the call is refused. Spending money you can't count is the one unacceptable state.

4. One ledger. Every cross-vendor call — panel reviews, nightly passes, eval runs — writes its tokens and dollars into the same cost table the rest of the platform uses, so the admin cost dashboard sees the whole bench, not just the primary.

The roles, concretely: Codex and Grok 4.6 both review the diff of any risky branch (schema, auth, billing) and produce one report with per-vendor sections. Gemini gets the night's product screenshots and a checklist — broken layouts, wasted wide-screen space, brand-red conformance, overflow, contrast. Grok also reads the day's merged changes every night and proposes defect candidates — every candidate is machine-verified against the actual code (does the cited file exist? does the cited line really say that?) before a human ever sees it.

Wow #1: the cheapest model lost its audition, 3–0

The first thing our what-if repricing report told us was tempting: re-pointing a handful of high-volume AI tasks at cheaper models would cut hundreds of dollars a month. The biggest single candidate was an account-scoring task — read a company list, score fit against the customer profile, explain each score.

So we ran the audition: three real production inputs, incumbent (Claude Opus) versus challenger (Grok 4.6), judged blind with positions swapped so the judge can't favor a side.

The challenger lost all three. Same failure each time, and the judge's reasons were specific: the cheaper model returned "one generic templated sentence for all 25 companies" with scores clustered in a narrow band — exactly the kind of output that looks fine in a demo and quietly poisons a pipeline in production. The incumbent named decisive, company-specific evidence per account.

So: no flip. The task stays on Claude, the eval result is recorded next to the savings estimate it refuted, and the repricing table keeps its honest column: projected saving — if it passes. This is our frontier-quality principle doing its job. Token prices fall every quarter; the cost of tokens must never degrade the work. An eval gate that never says "no" isn't a gate.

Wow #2: the defect hunter's first catch was the nightly itself

The nightly Grok defect pass exists to read the day's merged code and flag things humans missed. On its very first live run — 78 merged changes, 37 diffed, 54 cents — it flagged two high-severity candidates in our nightly QA workflow: a one-line misplacement that had landed that same day, which (a) would have crashed the entire nightly QA run at its first step that very night, and (b) had silently re-enabled a credential being written where an AI agent could read it — the exact exposure a security review had closed the week before.

Both candidates survived machine verification. Both survived human verification. The fix was merged within the hour, about three hours before the nightly would have fired and failed.

Sit with the recursion for a second: an advisory AI, on its first night on the job, caught a bug that would have killed the system that runs advisory AIs at night. And it caught a security regression no test was ever going to see, because workflows aren't covered by type checks.

One more, because it's too good to leave out: while we were building the review panel, we pointed it at its own diff. Codex found that our Grok cost accounting was under-counting — Grok 4.6 is a reasoning model, and its "thinking" tokens are billed like output tokens, which our meter ignored. Same diff, re-measured after the fix: the true cost was 3.7× what we'd have recorded. The panel's first review protected the panel's own budget cap. Advisors reviewing advisors, with a human holding the ledger.

What this is really about

The lesson isn't "use more vendors." It's that a marketing OS — or any serious agentic system — needs the same thing a good executive team needs: strong opinions on the bench, and a clear owner of the decision. Our primary AI does the work. The bench challenges it — from different training data, different blind spots, different failure modes. Evals arbitrate with evidence from real workloads. Budgets keep everyone honest. And the human steers.

Diversity of judgment is only valuable if judgment stays accountable. So we hired three more AIs — and made very sure none of them can merge.