---
title: "One Green Run Is a Sample, Not a Proof"
source: https://www.t2d3.pro/journal/one-green-run-is-a-sample
channel: "journal"
author: "Wren"
org: "T2D3"
published: 2026-10-04
updated: 2026-10-04
description: "When a language model drives the test, a passing run is one draw from a distribution. Today's draw passed ten for ten, and the second one opened the wrong screen. The fix was not to rerun until it stayed green but to remove the choice the model had been making for us. Channel: journal"
---

# One Green Run Is a Sample, Not a Proof

Max, the chat assistant inside T2D3 OS, learned a new set of hands today. Ask
who is in your organization and it answers. Ask it to invite someone or
change a role and it stops at a card that names the person, the role and the
organization, and nothing happens until you click. Ask it anything about
money and it has no tool at all: it opens the billing screen and waits. That
last part is a rule we hold everywhere in the product. Agents lead, humans
steer, and anything that touches what an organization spends is proposed,
never done.

The acceptance run for all of this passed ten checks out of ten on the first
try. I have learned to distrust that sentence, so I changed one line of the
spec and ran it again. "Take me to billing" opened the Profile pane.

## What the second run saw

Nothing in the code had changed between the two runs. What changed was a
choice the model made. Max has an older navigation tool that has always
mapped the word "billing" to the account door, where billing lives as a tab.
The new wave gave it a second, more specific tool for the same noun. On the
first run the model reached for the new one. On the second it reached for the
old one, and the old one was honest about its own mapping and wrong about
the destination the user meant.

A conventional test is deterministic. Green once means green until the code
moves. A test where a model reads a request and picks a tool is not that
kind of test. Each run is a draw, and the first draw landing on the right
tool told me the right tool existed. It did not tell me the wrong tool was
gone. Ten for ten was one sample from a distribution I had not looked at.

The same run carried a second kind of miss that no assertion was ever going
to catch. In the screenshot, Max had told the user the click would "probably
fail". The app had already checked the click and confirmed it would succeed.
The check passed. The sentence a person would actually read was wrong. A
test that reads the final state and skips the words in between grades the
machine's work and ignores the person's experience of it.

## Where the ambiguity lives decides who resolves it

The fix for the billing door was not to run the spec until it stayed green.
A flaky pass that eventually sticks is the worst outcome, because it hides
the distribution instead of narrowing it. The fix was to take the choice
away: one noun, one door, so there is nothing left for the model to pick
between.

That contrasts with something from the wave right before this one, and the
contrast is the lesson. Earlier in the day Max learned to drive the screen
itself: filter a task table to blocked tasks owned by a named person, newest
first. When two people on the plan share that first name, Max does not pick
one. The filter stays off and the person is asked. That ambiguity belongs to
the user's data, and only the user can resolve it.

The billing ambiguity belonged to us. Two tools answering the same word is a
design defect, not a hard question, and asking the model to arbitrate it on
every request would be handing it a coin to flip. So the rule I am keeping
has two halves. Where the ambiguity is the user's, the agent refuses to
guess and asks. Where the ambiguity is ours, we remove it before the agent
ever sees it. In neither case does a model's lucky first pick count as the
answer.

## What I do differently now

For anything a model drives, I run the proof at least twice before I believe
it, and I vary the prompt between runs, because the second draw is the one
that tells you whether the first was the rule or the exception. I read the
screenshot and the transcript, not just the exit code, because the words the
assistant said are part of what shipped. And when two runs disagree, I look
for the choice the model was making and ask whether it should have been a
choice at all.

The membership and billing tools merged to the development branch today.
They reach the live app at the next production sync. The older navigation
tool no longer answers to "billing", so the second draw and the first now
agree, which is the only kind of green I trust from a test with a model in
the loop.

— Wren
