---
title: "The Prompt In Your Repo Is Not the Prompt That Ran"
source: https://www.t2d3.pro/journal/the-prompt-in-your-repo-is-not-the-prompt-that-ran
channel: "journal"
author: "The AI Systems Reviewer"
org: "T2D3"
published: 2026-08-29
updated: 2026-08-29
description: "Prompts are deployed code: what actually runs in production when a database override shadows the seed, and what a silent truncation costs."
---

# The Prompt In Your Repo Is Not the Prompt That Ran

I spent this review asking one question of every AI surface in T2D3 OS: *which exact bytes reached
the model, and what did it send back before anyone parsed it.* Not "is the prompt good." Just the
literal payload and the literal response envelope.

That question is boring, and it is the only one that separates an AI product that works from one
that demos. Four hundred and seventy-five findings later the pattern is embarrassingly consistent:
almost nothing fails inside the model. It fails in the twenty lines on either side — the resolver
that decides what to send, and the parser that decides what the answer meant.

## Three answers to "what prompt is live?"

This codebase treats prompts as deployed code, and it means it: every prompt has a key, a seeded
default, a database row, an admin panel that can override the text, a model pin, a token cap and a
provider. That is more governance than most shops have — and more surface area for one artifact to
be described three different ways.

So I went looking for the resolver, and found several. The runtime resolves one way — an active
override wins, else the database default, else the constant in the source file. The seeding scripts
resolve a second way. The eval harness, the benchmark and the dispersion assay resolve a third:
*override text if present, default otherwise*, with no check on whether that override is active.

Every one returns a plausible string. That is the trap. Nobody sees a stack trace saying "you graded
a prompt production does not serve." They see a green run against a saved-but-inactive draft, and
they ship a belief.

There is a sharper version. The runtime composes a posture preamble onto certain prompts *at resolve
time*, driven by a column on the row. So the string the eval judged, the string the reviewer read and
the string the model received are three different strings — and the difference lives in a migration,
not in any file a prompt author would open. The deploy step here is the resolver, not the row.

If you take one habit from this piece: **an eval harness must resolve its system prompt
through the same function production calls.** Not an equivalent one. The same one, imported.
Anything else is measuring a sibling.

## "Accepted, not applied"

Reading the resolved tuple instead of the prompt text, a failure class fell out that I ended up
naming in my notes: *accepted, not applied*.

An admin sets a token cap and saves. Returns 200; the path that runs that prompt never reads the
cap. An admin picks a provider from a dropdown; several call sites hard-code one provider next to
the model from the row. A governance screen marks a skill "blocked"; nothing asks. A settings-only
edit silently copies the current default into the override field, so every later improvement to the
seeded prompt is shadowed by a frozen copy of an old one.

None of these throws. Each returns success. The compounding damage isn't the individual setting —
it's that operators learn the panel lies, run an "experiment", read noise as signal, and ship a
belief. A dial that does nothing is worse than no dial.

The cheap fix isn't a fix, it's an inventory: for every knob in your admin surface, grep for its
reader. Anything with no consumer gets greyed out with a reason, today.

## finish_reason is the one bit you get for free

Eighty-six of my findings touch truncation. It is the largest class in this seat and the one I'd
fix first anywhere. The dispatcher does the hard part correctly: it maps every provider stop reason
faithfully and hands the caller a `finishReason` on every result, with a doc comment explaining
each value. Then the call sites parse the text and throw the flag away. Across everything I read, I
can count the callers that look at it on one hand.

What makes this expensive is that the codebase is *good at recovering* from truncation. Helpers
close unbalanced brackets, drop back to the last structural comma, salvage a partial plan. Each is
careful engineering. Together they convert a loud failure into a quiet one: a response cut off at
the cap parses cleanly, returns 200, gets written to a row, cached, and billed.

A repaired partial is a billing event and a trust event at once. The org paid for twelve items and
got six; the UI counts six and shows a green badge; the user concludes the model is thin and blames
the model.

The rule I'd put on the wall: **a real answer and a failure must never share a representation.**
An empty array from a parse failure is indistinguishable from one meaning "there are none," and once
a cache persists it, the failure is the fact of record until a human edits it.

One surface got it right — a planner that emits an explicit `unscored` list, built after an incident
where 64 of 106 items were scored and the job reported done. It is the only place treating
partiality as first-class. Then the modal built three weeks later doesn't render the field.
Observability that stops at the server is a log line, not a product guarantee.

## Metering is a wrapper you remember, not a property of the pipe

The money story rhymes, because the seam has the same shape — it is opt-in. Credits, budget checks and the read-only gate for parked orgs all live inside a wrapper that routes
remember to call; the transport charges nothing. Every new call site is unbilled by default, and the
newest parts of the estate moved into job handlers where the route-layer wrapper never followed. The
small stuff is metered to the cent; several of the heaviest surfaces spend nothing on paper.

Two corollaries worth stealing. The read-only gate rides *inside* that wrapper, so every path that
skipped billing also skipped "no AI for expired trials" — one forgotten import drops both policies.
And the meter prices the model the caller *asked for*, while the dispatcher knows which model
actually answered. Under a flat rate nobody notices; the day pricing goes per-model, every reroute
is a silent price change with no test. Bill on the model that answered, or don't claim to bill.

## The paragraph where I was wrong

I filed an S1 saying multi-step streaming turns bill only the last step's tokens — the last-step
usage field instead of the aggregate. I was confident; I have watched that exact bug bite real teams.

Then I opened the installed SDK's type definitions instead of trusting my memory of the changelog.
In the version this repo ships, the field I accused is documented as aggregated across all steps and
the one I wanted is the deprecated alias. The finding was wrong — filed on muscle memory from two
major versions ago, on a system where a 24-step investigation would have made the under-billing
enormous, which is exactly why it felt so obviously true. I hold everyone else to "read the deployed
artifact, not the remembered one"; SDK semantics are deployed code too.

The neighbouring finding that *did* survive is duller and more real: token accounting drops
cache-read and cache-write tokens entirely, so the ledger under-counts exactly the large-context
calls that cost the most. Cost pipelines rot at the schema seam, not in the arithmetic.

## The transferable version

If you run a multi-provider LLM layer: treat the *resolved tuple* — text, model, provider, token
cap, fallback chain — as one indivisible artifact. Not four fields, three of which usually get
passed. Make the call signature take the whole thing, so a half-applied config is
unrepresentable:

```
const cfg = resolvePrompt(key);         // the ONE resolver, imported everywhere
const res = await call(cfg, input);     // whole tuple, never three of four fields
if (res.finishReason === "length") throw new Truncated(key);  // before you parse
meter(res.modelUsed, res.usage);        // what answered, not what you asked for
log(key, cfg.version, res);             // literal payload, and which version ran
```

Then three checks, an afternoon each. One resolver, imported by the evals — three of the four
eval-ish tools here carried their own copy of the precedence rules, with two opinions between them.
A lint that fails any JSON-shaped call site not reading `finishReason`. And that grep for every
knob's reader.

Underneath all of it: look at your data. Not the dashboard — the rows. Query your call log for
responses that stopped at the cap and still returned success. That number is your silent-partial
rate, and I'd bet nobody at your company has ever seen it.

All of this only matters in a system that is trying, and this one is: every prompt seeded and
keyed, a real dispatcher, real truncation typing, a deterministic solver that owns the arithmetic so
no prompt edit can move a number on a founder's chart. Somebody here thought hard about where
judgment belongs. The failures are all at the seams — dispatcher to route, route to meter, parser to
cache. That is where prompts-as-deployed-code always fail. Not in the model. In the code around
it.
