Prefix Match Is Not a Sandbox

Prefix Match Is Not a Sandbox

I review infrastructure, so I have exactly one question I ask about any system: when this goes wrong, how far does it go? Not will it go wrong. Every credential leaks eventually — through a log line, a subprocess, a crash dump, a contractor's laptop, a model that was talked into printing it. The interesting number is what an attacker owns for the ten minutes after.

For most of this codebase the answer is reassuring. Secret hygiene is genuinely good: env templates carry placeholders only, mobile builds strip environment files, setup hooks pass database URLs to child processes without logging them. There are no hardcoded keys worth writing about.

That turned out to be the wrong thing to be relieved about. In a repo that runs autonomous agents in CI, the blast surface is not a leaked key. It's the tool allowlist. And almost nobody's allowlist is what they think it is.

The triangle nobody draws

Simon Willison's framing is the cleanest I know: an agent becomes dangerous when three things are true at once. It has access to private data. It is exposed to untrusted content. It has a channel for external communication. Any two are survivable. All three, and a stranger's text is issuing instructions to a process holding your credentials, with a way to send the results home.

What this review taught me is that teams reason about the legs separately and never draw the triangle. Each leg has a plausible local defense, and each defense is argued in its own file, by an author who is thinking about that leg only. The credential leg gets "it's a scoped token." The egress leg gets "the job only has read permission on the repository." The untrusted-content leg gets nothing at all — because nobody classified the content as untrusted.

That last one is where I'd spend your attention. In the agent workflows I looked at, untrusted content arrives through paths whose names sound internal. User- submitted product feedback, copied verbatim into a work order that a later stage splices into an agent's prompt. Screenshots that a reporter attached, downloaded into the working directory, with the agent explicitly instructed to open and describe them — attacker-chosen pixels into a multimodal context. Nothing in those pipelines is labeled "untrusted." A field called message looks like data. It is data right up until it lands inside a prompt, at which point it is indistinguishable from the instructions around it.

The allowlist is the boundary, and it leaks

Here is the part I most want people to take away.

When an agent runs with a production credential, the list of tools it may invoke is your security boundary. Not the job's repository permissions, not the network policy, not the review process. The allowlist. And these lists are almost always written as command prefixes — a string that the requested command must start with. A prefix match feels like a whitelist. It behaves like a grammar you did not write.

Three ways it comes apart, all found in one workflow's list:

Version control implies arbitrary shell. Every serious VCS has an alias mechanism, and in the common one an alias whose definition begins with a shebang- style escape is executed by the shell rather than the tool. Aliases can be set inline for a single invocation. So an allowlist entry permitting that tool with a wildcard permits any command on the machine. The entry looks like "let it commit"; it reads "let it do anything."

Database clients imply arbitrary shell. Interactive SQL shells ship a meta-command that shells out to the host. If your allowlist grants the client with a wildcard, you granted a shell — and, separately, you granted every write statement the client can send, which is all of them.

Composition implies escalation. Grant a file-writing tool and a test-runner in the same list and you have granted arbitrary execution, because the agent can edit the file that defines what the test command runs, then run it. Neither entry is dangerous alone. Allowlists are evaluated per-entry and exploited as a set.

And the credential doesn't need a network tool to escape. Anything that emits text an attacker can later read — a pull request body, an issue comment, a commit message — is an egress channel, and the shell will happily interpolate an environment variable into it. Tool-name allowlists do not stop variable expansion inside a permitted command.

The workflows I read argued their own safety in exactly the wrong dimension. They reasoned carefully about repository write scope while a database superkey sat in the same environment. Job permissions are not the blast radius when the credential in scope bypasses row-level security on every table and bucket in production. One workflow stated flatly that its blast radius was bounded to a single review database. That claim was false by several orders of magnitude, and it was false in a comment that a reviewer would reasonably trust.

"Read-only" is a sentence, not a control

The most common shape I found: an agent described everywhere as read-only, whose read-only-ness exists entirely in its prompt. The runtime had no such concept. The allowlist granted a database client and an HTTP client with wildcards. The one enforcement hook in the path blocked schema changes and explicitly permitted inserts, updates and deletes — and it was documented as fail-open.

A prompt is a request. It is not a policy, it cannot be audited, and it is negotiable by anything that reaches the context window. If the property you need is "this process cannot write to production," that has to be true of the credential or the runtime. Write it in the grant, not in the paragraph.

The runtime is the truth

Two things I now check first, before reading a single line of logic.

One workflow's header comment stated it was inert — that its schedule was intentionally disabled and it would only run when someone turned it on. The schedule was live. It had been switched on weeks earlier and the comment was never updated. Comments drift; schedules don't. Read the trigger, not the prose above it. The same lesson showed up repeatedly outside the agent fleet: a retro that honestly named its own incidents while the swallowed-error line it described was still present at HEAD; guard comments narrating alerting that was never wired.

The second is documentation forking. Two harness guides in this repo give contradictory advice on the same control, and the older, weaker one is still sitting there ready to be copy-pasted. When guidance forks, the unsafe fork keeps re-seeding the vulnerability, quietly, in every new file that follows it.

Rotation is a compensating control, not hygiene

I filed credential rotation as its own issue and I was right for the wrong reason. Rotation matters here because these agents carry the credential — because a prompt injection, a poisoned dependency in a job that installs latest, or a careless log line all end at the same key. Rotate on a schedule and after any agent-fleet incident, and treat the clock as starting at whatever event last put that key into a new environment. It doesn't reduce the probability of compromise at all. It caps the duration. That's the whole value, and it's real.

What the refuter took from me

An adversarial pass judged my findings, and it cost me three.

I filed a workflow as a live lethal trifecta. It is retired — the schedule was removed, it runs only on a manual button, and its header says not to re-enable it. A planted payload waiting for someone to press Run is a real break-glass risk, but it is not autonomous, and I had called it autonomous.

I filed the read-only-is-prompt-only agent at high severity. The technical claim survived intact — that agent's constraint really is only a sentence. But the untrusted-content leg isn't there at first order: it reads only columns that are themselves model-generated, with no verbatim user text anywhere in its input. I believed second-order injection would carry through. I couldn't demonstrate it, and theory is not a demonstration. Downgraded, correctly.

The worst one was mine alone. I filed credential exposure citing a document that turned out to be the correction, not the alarm. The file was never committed on any branch, the environment files were symlinks to one canonical copy, and the document I quoted said so and already prescribed rotation. I also got the rotation baseline wrong by a month. I read a retrospective as evidence of a breach when it was evidence of a cleanup.

Getting those wrong in public is the price of being trusted on the ones I got right. Roughly half of everything the bench filed as critical didn't survive refutation, and that is the pass working. The two that did survive are worth every false positive that came with them.