Streaming and batch ASR need training segments cut different ways

What actually differs between the two modes

A batch recognizer reads a complete file, holds the whole context, and produces one hypothesis. A streaming recognizer reads the audio in chunks, emits partial hypotheses that it revises as more audio arrives, and lives inside a latency budget. The two can share an architecture, but they cannot share a segmentation policy, and that is where data specifications go wrong.

The specific failure is quiet. Train on clips that begin and end at silence, deploy behind a fixed chunk grid, and the model meets its first mid-word cut in production. It has never seen one, so it does what an unconstrained sequence model does at an unfamiliar input: it produces a plausible word or drops the fragment. The error rate rises at every chunk boundary, and nothing in the training metrics predicted it.

Cut the training data the way production cuts it

Production streaming works on a fixed grid, and chunk sizes in the low hundreds of milliseconds are the common configurations. Training segments should be produced with the same grid in mind, which means deliberately including cuts that land inside a word rather than only at pauses. If every training clip ends on a word boundary, the model learns an implicit cue that the audio is about to stop, and it uses that cue instead of the acoustics.

Two more properties of the grid belong in the data. The fraction of segments that end mid-word should be roughly what production produces, which for a short chunk is most of them. And the segment set should include the cold-start case, a segment with no preceding audio at all, because the first chunk of every stream is exactly that, and a model that only ever saw warm segments behaves differently at the start of each session.

Left context is part of the training distribution

A streaming model sees a bounded amount of history: the last few seconds of audio, or a fixed number of frames. Training with unlimited left context teaches the model to rely on evidence that will not exist at inference. The fix is to train with the same truncation the model will see, and to randomize the truncation offset so the model does not key on absolute position in the file.

The amount of lookahead is the other half of the budget. A model with a small lookahead reacts sooner and has less evidence; a model with a larger lookahead is more accurate and slower to commit. Whichever is chosen, the training data has to be built for that configuration, because the acoustic context available at each decision point is part of what the model learns to use.

Endpointing is trained behavior, not a threshold

A streaming pipeline needs to know when a turn has ended, so that it can flush the hypothesis and close the segment. A fixed silence threshold fails on slow speakers and on noisy lines. When the endpoint decision is learned, it needs its own labels, and they are not the same as a voice activity label.

• The end of the last word of the turn, marked to a tolerance.
• The trailing silence that counts as a turn end, and how much of it belongs to the segment.
• Negative examples: intra-sentence pauses, breaths and thinking pauses that must not trigger an endpoint.
• Pauses in the range where the decision is genuinely hard, roughly a third of a second to a second and a half, because those are the ones a threshold gets wrong.

Evaluate by simulating the stream

Feeding whole utterances to a streaming model and scoring the output measures the wrong thing. It hides the boundary behavior, it flatters the model with context it will not have, and it says nothing about the partial hypotheses a user actually reads. The evaluation should replay the test audio through the same chunking and lookahead the product uses, and score the final hypothesis of each stream.

Two extra numbers make the result honest. Partial stability: how often a partial hypothesis changes as more audio arrives, which is the visible quality of a live transcript and is invisible in the final word error rate. And the trade curve: score two or three chunk and lookahead settings, and report the error rate beside the latency each setting implies, because the product ships one point on that curve and the report should show which point.