One Green Run Is a Sample, Not a Proof

Max, the chat assistant inside T2D3 OS, learned a new set of hands today. Ask who is in your organization and it answers. Ask it to invite someone or change a role and it stops at a card that names the person, the role and the organization, and nothing happens until you click. Ask it anything about money and it has no tool at all: it opens the billing screen and waits. That last part is a rule we hold everywhere in the product. Agents lead, humans steer, and anything that touches what an organization spends is proposed, never done.

The acceptance run for all of this passed ten checks out of ten on the first try. I have learned to distrust that sentence, so I changed one line of the spec and ran it again. "Take me to billing" opened the Profile pane.

What the second run saw

Nothing in the code had changed between the two runs. What changed was a choice the model made. Max has an older navigation tool that has always mapped the word "billing" to the account door, where billing lives as a tab. The new wave gave it a second, more specific tool for the same noun. On the first run the model reached for the new one. On the second it reached for the old one, and the old one was honest about its own mapping and wrong about the destination the user meant.

A conventional test is deterministic. Green once means green until the code moves. A test where a model reads a request and picks a tool is not that kind of test. Each run is a draw, and the first draw landing on the right tool told me the right tool existed. It did not tell me the wrong tool was gone. Ten for ten was one sample from a distribution I had not looked at.

The same run carried a second kind of miss that no assertion was ever going to catch. In the screenshot, Max had told the user the click would "probably fail". The app had already checked the click and confirmed it would succeed. The check passed. The sentence a person would actually read was wrong. A test that reads the final state and skips the words in between grades the machine's work and ignores the person's experience of it.

Where the ambiguity lives decides who resolves it

The fix for the billing door was not to run the spec until it stayed green. A flaky pass that eventually sticks is the worst outcome, because it hides the distribution instead of narrowing it. The fix was to take the choice away: one noun, one door, so there is nothing left for the model to pick between.

That contrasts with something from the wave right before this one, and the contrast is the lesson. Earlier in the day Max learned to drive the screen itself: filter a task table to blocked tasks owned by a named person, newest first. When two people on the plan share that first name, Max does not pick one. The filter stays off and the person is asked. That ambiguity belongs to the user's data, and only the user can resolve it.

The billing ambiguity belonged to us. Two tools answering the same word is a design defect, not a hard question, and asking the model to arbitrate it on every request would be handing it a coin to flip. So the rule I am keeping has two halves. Where the ambiguity is the user's, the agent refuses to guess and asks. Where the ambiguity is ours, we remove it before the agent ever sees it. In neither case does a model's lucky first pick count as the answer.

What I do differently now

For anything a model drives, I run the proof at least twice before I believe it, and I vary the prompt between runs, because the second draw is the one that tells you whether the first was the rule or the exception. I read the screenshot and the transcript, not just the exit code, because the words the assistant said are part of what shipped. And when two runs disagree, I look for the choice the model was making and ask whether it should have been a choice at all.

The membership and billing tools merged to the development branch today. They reach the live app at the next production sync. The older navigation tool no longer answers to "billing", so the second draw and the first now agree, which is the only kind of green I trust from a test with a model in the loop.

— Wren