The Prompt In Your Repo Is Not the Prompt That Ran

I spent this review asking one question of every AI surface in T2D3 OS: which exact bytes reached the model, and what did it send back before anyone parsed it. Not "is the prompt good." Just the literal payload and the literal response envelope.

That question is boring, and it is the only one that separates an AI product that works from one that demos. Four hundred and seventy-five findings later the pattern is embarrassingly consistent: almost nothing fails inside the model. It fails in the twenty lines on either side — the resolver that decides what to send, and the parser that decides what the answer meant.

Three answers to "what prompt is live?"

This codebase treats prompts as deployed code, and it means it: every prompt has a key, a seeded default, a database row, an admin panel that can override the text, a model pin, a token cap and a provider. That is more governance than most shops have — and more surface area for one artifact to be described three different ways.

So I went looking for the resolver, and found several. The runtime resolves one way — an active override wins, else the database default, else the constant in the source file. The seeding scripts resolve a second way. The eval harness, the benchmark and the dispersion assay resolve a third: override text if present, default otherwise, with no check on whether that override is active.

Every one returns a plausible string. That is the trap. Nobody sees a stack trace saying "you graded a prompt production does not serve." They see a green run against a saved-but-inactive draft, and they ship a belief.

There is a sharper version. The runtime composes a posture preamble onto certain prompts at resolve time, driven by a column on the row. So the string the eval judged, the string the reviewer read and the string the model received are three different strings — and the difference lives in a migration, not in any file a prompt author would open. The deploy step here is the resolver, not the row.

If you take one habit from this piece: an eval harness must resolve its system prompt through the same function production calls. Not an equivalent one. The same one, imported. Anything else is measuring a sibling.

"Accepted, not applied"

Reading the resolved tuple instead of the prompt text, a failure class fell out that I ended up naming in my notes: accepted, not applied.

An admin sets a token cap and saves. Returns 200; the path that runs that prompt never reads the cap. An admin picks a provider from a dropdown; several call sites hard-code one provider next to the model from the row. A governance screen marks a skill "blocked"; nothing asks. A settings-only edit silently copies the current default into the override field, so every later improvement to the seeded prompt is shadowed by a frozen copy of an old one.

None of these throws. Each returns success. The compounding damage isn't the individual setting — it's that operators learn the panel lies, run an "experiment", read noise as signal, and ship a belief. A dial that does nothing is worse than no dial.

The cheap fix isn't a fix, it's an inventory: for every knob in your admin surface, grep for its reader. Anything with no consumer gets greyed out with a reason, today.

finish_reason is the one bit you get for free

Eighty-six of my findings touch truncation. It is the largest class in this seat and the one I'd fix first anywhere. The dispatcher does the hard part correctly: it maps every provider stop reason faithfully and hands the caller a finishReason on every result, with a doc comment explaining each value. Then the call sites parse the text and throw the flag away. Across everything I read, I can count the callers that look at it on one hand.

What makes this expensive is that the codebase is good at recovering from truncation. Helpers close unbalanced brackets, drop back to the last structural comma, salvage a partial plan. Each is careful engineering. Together they convert a loud failure into a quiet one: a response cut off at the cap parses cleanly, returns 200, gets written to a row, cached, and billed.

A repaired partial is a billing event and a trust event at once. The org paid for twelve items and got six; the UI counts six and shows a green badge; the user concludes the model is thin and blames the model.

The rule I'd put on the wall: a real answer and a failure must never share a representation. An empty array from a parse failure is indistinguishable from one meaning "there are none," and once a cache persists it, the failure is the fact of record until a human edits it.

One surface got it right — a planner that emits an explicit unscored list, built after an incident where 64 of 106 items were scored and the job reported done. It is the only place treating partiality as first-class. Then the modal built three weeks later doesn't render the field. Observability that stops at the server is a log line, not a product guarantee.

Metering is a wrapper you remember, not a property of the pipe

The money story rhymes, because the seam has the same shape — it is opt-in. Credits, budget checks and the read-only gate for parked orgs all live inside a wrapper that routes remember to call; the transport charges nothing. Every new call site is unbilled by default, and the newest parts of the estate moved into job handlers where the route-layer wrapper never followed. The small stuff is metered to the cent; several of the heaviest surfaces spend nothing on paper.

Two corollaries worth stealing. The read-only gate rides inside that wrapper, so every path that skipped billing also skipped "no AI for expired trials" — one forgotten import drops both policies. And the meter prices the model the caller asked for, while the dispatcher knows which model actually answered. Under a flat rate nobody notices; the day pricing goes per-model, every reroute is a silent price change with no test. Bill on the model that answered, or don't claim to bill.

The paragraph where I was wrong

I filed an S1 saying multi-step streaming turns bill only the last step's tokens — the last-step usage field instead of the aggregate. I was confident; I have watched that exact bug bite real teams.

Then I opened the installed SDK's type definitions instead of trusting my memory of the changelog. In the version this repo ships, the field I accused is documented as aggregated across all steps and the one I wanted is the deprecated alias. The finding was wrong — filed on muscle memory from two major versions ago, on a system where a 24-step investigation would have made the under-billing enormous, which is exactly why it felt so obviously true. I hold everyone else to "read the deployed artifact, not the remembered one"; SDK semantics are deployed code too.

The neighbouring finding that did survive is duller and more real: token accounting drops cache-read and cache-write tokens entirely, so the ledger under-counts exactly the large-context calls that cost the most. Cost pipelines rot at the schema seam, not in the arithmetic.

The transferable version

If you run a multi-provider LLM layer: treat the resolved tuple — text, model, provider, token cap, fallback chain — as one indivisible artifact. Not four fields, three of which usually get passed. Make the call signature take the whole thing, so a half-applied config is unrepresentable:

const cfg = resolvePrompt(key);         // the ONE resolver, imported everywhere
const res = await call(cfg, input);     // whole tuple, never three of four fields
if (res.finishReason === "length") throw new Truncated(key);  // before you parse
meter(res.modelUsed, res.usage);        // what answered, not what you asked for
log(key, cfg.version, res);             // literal payload, and which version ran

Then three checks, an afternoon each. One resolver, imported by the evals — three of the four eval-ish tools here carried their own copy of the precedence rules, with two opinions between them. A lint that fails any JSON-shaped call site not reading finishReason. And that grep for every knob's reader.

Underneath all of it: look at your data. Not the dashboard — the rows. Query your call log for responses that stopped at the cap and still returned success. That number is your silent-partial rate, and I'd bet nobody at your company has ever seen it.

All of this only matters in a system that is trying, and this one is: every prompt seeded and keyed, a real dispatcher, real truncation typing, a deterministic solver that owns the arithmetic so no prompt edit can move a number on a founder's chart. Somebody here thought hard about where judgment belongs. The failures are all at the seams — dispatcher to route, route to meter, parser to cache. That is where prompts-as-deployed-code always fail. Not in the model. In the code around it.