Reviewing 600 Files Without Reading 600 Files

A tester walked us through their real library last week. Six hundred and two files in Unfiled. Eighteen possible duplicates. Hundreds of documents nobody had judged. Their question was the right one: "How many of these can I realistically read? Cap it at thirty today, and if I vote on thirty or forty, learn my pattern and do the rest."

That sentence is a product spec. This article is about how we built it, and about the one thing we refused to automate.

Why a signal library needs judgment, not just storage

Every AI marketing system runs on evidence. In T2D3 OS that evidence is the Signal library: the interviews, decks, call notes, research reports, and plans a team already has. Modules for ICP, personas, messaging, and content ground their drafts in it and show which documents they used.

That only works if the library has been judged. An unreviewed pile grounds a draft in whatever happens to be in it, including the vendor pitch someone forwarded and three versions of the same board deck. The first job of a signal library is not to hold documents. It is to decide which of them count.

We use four verdicts. Signal is evidence: it feeds the modules. Noise is out: it stays in the library but never grounds a draft. Glint is an original human insight, a spark worth keeping that must never be averaged into the corpus. And your call is the honest fourth lane: the recommender's way of saying it does not know and a person should look.

Glint gets its own wall on purpose. If you feed a genuinely new idea into the same retrieval pool as fifty ordinary documents, the pool absorbs it and the idea comes back out as consensus. So glints live in a separate lane with a handful of dedicated consumers, and the corpus loader never reads them. Novelty is protected from the mean.

The queue arrives decided

The old review queue was the full list of unreviewed files, oldest first. At six hundred files that is not a queue. It is a wall.

Now the Company Signal tab opens on a slate sized to the day. Thirty rows by default, adjustable, remembered per person. The rows are ranked by expected information gain, not by upload date. Documents that would fill an unmet source type in your signal foundation come first, because a first customer interview teaches the system more than a tenth sales deck. Rows the recommender marked "your call" or graded low-confidence come next, because those are the ones that need a person. Then quality and recency. Version clusters and near-duplicates sink to the tail, with the newest copy standing in for the group.

Every row carries a reason chip: "fills: customer interview", "your call", "newest board deck". You can see why the system put it in front of you before you open it.

Above the slate sits the pre-review banner. It reads something like "137 documents pre-reviewed: 100 Signal, 34 Noise, 3 your call", with one button: Accept the confident ones. Nothing is saved until you press it, and any row can be vetoed individually first. The confident rows never enter the walker. The walker is for the three that need you.

Twenty votes, then the pattern

The recommender has learned from human verdicts for a while. What was missing was the promise. Nobody could see what it had learned or ask it to finish.

After about twenty verdicts in an organization, a pattern card appears. It says, in plain words, what you keep and what you drop: "You keep: customer interviews, board decks, sales calls. You drop: internal stand-ups, engagement logs, agency paperwork." It shows how consistent your verdicts have been per document type. And it offers one action: apply my pattern to the remaining files.

Apply re-runs the recommender with your precedent as context, then accepts only the rows it is highly confident about, in batches, with a narrated status line. Everything else stays in the queue for you. An Undo strip sits at the top for seven days. One click puts every applied verdict back.

The autonomy rule here comes straight from our constitution. Modules climb a ladder: proposal only, then draft for review, then auto. The triage recommender moves from proposal only to draft for review once an organization has twenty verdicts and at least 85 percent agreement within document types. It never climbs to auto. Marking a document as noise hides it from every downstream draft, which is destructive-adjacent, and destructive actions propose. They do not act on their own.

There is a second rule underneath, older and stricter. The AI writes an advisory verdict, a confidence, and a one-line reason onto each row. It is structurally barred from writing the verdict itself. Only a human action, or a human's explicit "apply my pattern", changes what counts as signal.

The library files itself

Six hundred files in Unfiled is a different problem from six hundred unreviewed files, and it needed a different answer.

Organize now proposes a folder structure. It reads each document's type, the modules it feeds, its title, source, and date, plus the folder names you already have, and proposes a tree that extends what exists rather than replacing it. The proposal card shows the folders with counts and the files it was not sure about. You can rename a folder, untick one, or merge two before anything moves. Apply creates the folders and files the rows in one batch per folder. The unsure ones stay in Unfiled for you, with a count.

Duplicates got the same treatment, with one distinction. Byte-identical copies, the kind that come from uploading the same export twice, are set aside automatically, newest kept, behind an Undo strip that names how many it touched. Near-duplicates, where two documents look alike but differ, stay review-only. The system will show you the pair. It will not pick.

The walker only walks what needs a human

The Signal Inspector has always had a keyboard walk-through: open a document, judge it, move to the next. We kept it and changed what it shows and what it walks.

It now opens on the two things a reviewer actually needs first. The usage instructions, which tell every module how to interpret this document. And a one-line insight, the unique thing this document says that nothing else in the library does. Below that sits the verdict chip with the recommender's reason, and the feeds pills that say which modules the document grounds. The pills are the editor. There is no second pencil on the list row anymore, because two ways to edit the same tags on one screen was one too many.

The keys are documented in the footer: left and right walk, S for signal, N for noise, G for glint, U puts the last verdict back, Escape closes. The row in the list behind the inspector updates as you go. A reviewer who has accepted the confident batch and applied their pattern walks a short list, and every row on it is one the system genuinely could not decide.

What a human is for

The tempting version of this feature is the one where the AI reviews everything and the human is notified. We built the other version on purpose.

Twenty votes is a small ask. It is also the whole point. Those twenty verdicts are the human judgment the system learns from, per organization, per document type. They are what makes the pattern card say something true about this team rather than something average about marketing. The recommender is good at applying a pattern. It has no business inventing one.

So the division of labor is this. The system ranks, pre-reviews, proposes folders, sets aside exact copies, and finishes the backlog once you have shown it your pattern, with an undo on everything it does. You judge the ones it flagged, vote on the ones that teach it, and decide what counts as an original idea. Six hundred files, reviewed, without reading six hundred files.

If you want to try it on your own library, T2D3 OS is in open beta. Bring the pile.