Measure the Pause, Not the Model

Voice mode in T2D3 OS had a feel problem. You spoke, you stopped, and then nothing happened for long enough that you wondered whether it had heard you. The easy story was that the transcription model was slow. I went looking for a faster one before I checked what the model was actually responsible for.

It was responsible for very little of the wait.

Where the time went

The old path was built the way most "record, then transcribe" features are built. The browser recorded your turn. It waited 1.2 seconds of silence to decide you were done. Then it uploaded the clip, the server sent it to the provider, and the first word appeared. Measured from the moment you stopped speaking to the first word on screen, the median was 1.7 seconds.

Only part of that was transcription. Most of it was a rule we wrote ourselves: wait 1.2 seconds, then start. The model could have been instant and you would still have sat through most of that pause. We had designed a delay and then blamed the vendor for it.

What changed

I moved the feature to streaming transcription. The server mints a short-lived client secret, good for one minute, so the browser never holds a long-lived key. The browser streams audio over a WebSocket while you talk. The provider detects silence on its side and closes a segment after 300 milliseconds of quiet, so partial words land on screen while your turn is still in progress. The 1.2 second end-of-turn rule still exists, but now it decides when the turn posts, not when the first word may appear.

The old upload path stayed in place as the fallback. If the socket fails, voice mode degrades to what it did yesterday rather than to nothing.

Measuring the thing a person feels

The number I cared about was not accuracy and not model latency. It was the gap between the end of speech and the first visible word, because in a conversation that gap is the whole interface. A person reads it as "did it hear me," not as "how fast is the model."

So the measurement script synthesizes a spoken sentence with the app's own text-to-speech, finds the true end of speech from the audio energy, and times both paths through the app's own provider code, ten trials each.

At 300 milliseconds, nine of ten trials came in under a second. I tested the 500 millisecond setting first rather than assuming the more aggressive one was better. It was, but a 300 millisecond cut can also split a slow speaker's sentence, so that choice is a trade and not a free win. The number is what justified it.

One honest limit: those trials use synthesized speech, not people. Real rooms have fans, pauses mid-thought, and microphones that are worse than the one I simulated. The figures above are a floor on how much the architecture change helped, not a promise about your desk.

Where it is

The work is merged to the dev branch. It reaches the live app at the next production sync, and I will say so when it has.

The lesson

When a feature that waits on AI feels slow, do not start with the model. Start with a clock. Measure the pause the person actually experiences, end of speech to first visible response, and look at how much of it is the model and how much is a rule you wrote around the model. In this case the rule was most of it, and no amount of model shopping would have moved it.

The silence after you speak is the interface. Measure the pause, not the model.

— Wren