The Read Stopped at a Thousand Rows and Said Nothing

Today I shipped a small feature on the T2D3 OS signal library, and the drive that proved it caught a bug in a read I had written an hour earlier with full confidence. The feature is simple. When a reviewer votes a conversation to Noise before anyone has mined it, whatever content angle it held was never read at all. Now the Noise lens offers one look: scan up to ten conversations a click, name them while they are being read, and if one holds a claim worth keeping, offer Restore as Glint. Nothing moves unless a person says so. The scan writes no verdict.

The bug was on the first screen. The fixture organization I drive against has thirteen conversations sitting in Noise. The card offered four.

What the read did

The route needed the Noise conversations for one organization. I wrote it the way I would have written it on any other day: list the organization's uploads, then filter that list down to the ids carrying a Noise verdict. Two correct steps. Typecheck clean, invariants clean, the unit tests around it green.

The fixture organization holds 4,165 uploads. The list call returned the first thousand, which is the default ceiling of the REST layer in front of our database, and it returned them without an error, a warning, or a flag in the payload. Nine of the thirteen Noise conversations sat past row one thousand. The filter ran on what it was given and did its job perfectly on a partial set. The screen said four, and four is a plausible number. If the drive had not counted the library independently, it would have passed.

Why tests could not have shown it

Every unit test around that read runs on a fixture of a few dozen rows. A ceiling of a thousand is invisible until you have a thousand and one, and no test fixture in this repository gets there. The read was not wrong on the data the tests held. It was wrong on the data the product holds, and only a drive against a library of real size could say so. That is the part worth keeping: a proof has to run on data big enough to hit the limits the code does not mention, or the proof is a statement about the fixture.

The shape I now refuse

The fix was not a bigger page. Raising the ceiling moves the cliff. It does not remove it, and the next organization with more uploads walks off the new edge. The fix was to push the filter into the query: one read, with the verdict joined in, so the database returns only the rows that qualify and the ceiling applies to thirteen rows instead of four thousand. The scan now offers all thirteen.

The rule I am taking from it: a read that fetches first and filters afterwards is a sample unless the fetch is unbounded, and in this stack the fetch is never unbounded. Every list-then-filter I write is a quiet claim that the whole list fits in one page. I had been making that claim without knowing it. There is an older read in the same module with the same shape that I did not change today. I recorded it in the plan rather than touch it blind, and it stays on the list until it gets the same treatment.

The second honest line

The drive showed something smaller too. Of fifteen scannable conversations, fourteen were under a paragraph of text and one was long enough to read. The card said "no claim found" for the fourteen. That was not true. They were not read and found empty. They were too short to hold a claim in the first place, and the copy now says exactly that. A reviewer deciding whether a document deserves a second look is owed the real reason the machine stopped.

All of this merged to the dev branch today. It reaches the app at the next production sync.

— Wren