Silence and non-speech in training audio: what to cut and what to keep

A model that never heard silence invents words

The failure is easy to reproduce. Take a recognizer trained on segments where speech fills the clip, feed it a music track or a minute of room tone, and it will not return nothing. It returns fluent text, because the decoder has no option for emptiness and the language model is confident about fluent sentences. The same model inserts words over long pauses inside otherwise ordinary utterances.

The cause sits in the training data rather than in the architecture. If every segment contains speech, the model never observes the case where the correct output is nothing. A deliberate non-speech portion, with empty or tagged transcripts, is what teaches that option. The reverse mistake exists too: a dataset that is mostly silence teaches the model to prefer no-speech and to drop quiet speech, so the share is a decision to be made, not a side effect of trimming.

Trim the outer silence to a bounded pad, keep the pauses inside

Trimming policy has two parts and they are usually conflated. The outer silence of a clip — the seconds before the first word and after the last — gets cut back to a fixed pad, the same length every time. Unbounded outer silence wastes compute, skews any duration-based batching, and makes the clip duration a poor description of its content. The pad has to be long enough that the first and last phonemes survive, which is where the next section comes in.

The pauses inside an utterance are a different matter and they stay. Removing them changes the timing and the prosody, breaks the correspondence between the audio and the transcript, and teaches the model that speech never pauses. A set of clips that keep their full natural leading silence is also worth holding onto, because those are the examples that teach endpointing, and a corpus that has removed every long silence has removed the training signal for the moment when a system should stop listening.

Trimming is not neutral: the first and last phonemes pay

An energy threshold or a voice activity detector does not cut silence. It cuts wherever the signal crosses a threshold, and the phonemes at word edges live near that line. A plosive has a silent closure before its burst, and an aggressive trim removes the closure and sometimes the burst itself. A final fricative is low-energy and disappears under a threshold set for vowels. The result is a corpus with systematically damaged word edges, and a model that appears to have a weakness at word boundaries.

The protections are cheap. Padding of tens to a couple of hundred milliseconds around the detected boundaries keeps the edges intact. A short fade at the cut point avoids the click that a hard cut at a nonzero sample produces, and the cut should land at a low-energy point or a zero crossing rather than anywhere in the waveform. The check is by ear on a sample, listening to the first and last fraction of a second of the trimmed clips, which is exactly where the damage concentrates.

Sort non-speech into categories and decide each one

A single noise bucket hides the decisions that matter, because the categories behave differently. Background music with vocals is the strongest hallucination trigger. Other people talking while the primary speaker is also talking is overlap, a problem with its own solutions. Television and radio are speech-like audio from a speaker who is not the subject of the recording. Environmental noise — traffic, keyboards, wind, appliances — behaves more like silence. Human non-speech sounds such as laughter and coughs usually belong with the speech clips, tagged.

Each category gets a decision in the specification: keep with an empty transcript, keep with a tag, or drop. The test for the decision is the deployment audio. A policy that removes every music clip and every television clip trains a model for a world that does not exist, and the first real-world recording with a radio in the background is where that shows up. If a category is dropped from training, it should still appear in the test set, so the weakness is measured rather than hidden.

Build the non-speech slice into the test set

The evaluation that catches this class of failure is small and specific: a slice of silence, music, environmental noise and distant speech, scored by what the model outputs. The headline number is the share of clips where it emits text at all, the hallucination rate, and for speech clips with long pauses, whether it inserts words at the pause.

The spec fields that make the policy auditable: the pad length and how it was applied, the fade or zero-crossing rule at cut points, the non-speech categories that were kept and what their transcripts contain, and the size of the non-speech portion expressed as a share of the speech hours. That last field is the one buyers never see and the one that explains most of the difference between two datasets that both describe themselves as clean speech.