The preprocessing pipeline for ASR: what each step does and why the order matters

Raw recordings arrive with everything wrong at once: silence at both ends, two channels, a rate that depends on the device, a level that depends on how close the speaker sat to the microphone, and a chair creak somewhere in the middle. The preprocessing pipeline is the sequence of steps that turns that into something a recognizer can train on.

The sequence is not arbitrary. Each step changes what the next one sees, and a few of the steps are irreversible once they are done wrong. What follows is one order that holds up in practice, the reason each step sits where it does, and the check that catches the common failure at that step.

Decode once, at the native rate

If the source is FLAC, OGG, or a compressed format, decode it to PCM once and keep that copy. Decoding is not neutral: lossy decoders differ in how they pad the start of the file and how they handle stereo, so two machines decoding the same MP3 can produce arrays that differ by a few hundred samples. Do the conversion one time, record the tool and its version in a manifest, and never chain a decode into a re-encode.

• Check: soundfile.info(path) reports the rate, channel count, and subtype you expect, and frames divided by rate matches the duration your inventory claims.
• Check: sample values stay inside the range from -1.0 to 1.0. A file that exceeds it was scaled somewhere upstream.
• Check: no file reports a rate of zero or a frame count of zero. That is what a truncated upload looks like.

Trim silence before resampling

Silence trimming comes next because it is cheap and because it makes the level measurement in a later step honest. A file that is forty percent silence has an average level dragged down by the silence, and normalizing that average pushes the speech too loud. Use a short-frame energy threshold or a voice activity detector to find the first and last speech frames, keep about 0.1 seconds of padding on each side so the onset is not clipped, and drop the rest.

The same pass should drop files that turn out to be empty. A recording where no frame crosses the threshold is either a broken take or an upload that never completed. Both should leave the pipeline here, with a count of how many were dropped, rather than arriving at training as a stretch of noise.

Resample to the training rate, once

Downsampling needs an anti-aliasing filter, and the fastest settings of some libraries are not good enough for a corpus. In librosa, pass res_type equal to soxr_hq. In scipy, use resample_poly with the integer ratio rather than resample: 48 kHz to 16 kHz is up=1, down=3. A 48 kHz file resampled with a weak filter keeps energy above 8 kHz and folds it back into the speech band, where it is indistinguishable from a lisp.

The rate you resample to should be the rate your model consumes. Pick it once, in one place in the code, and make it a constant that both the preprocessing script and the training configuration read. When those two disagree, the failure is quiet: the model trains, the loss goes down, and the accuracy is simply lower than it should be.

Normalize level last, and only once

Loudness normalization belongs at the end, after the signal is final, because resampling slightly changes peaks and trimming changes the average. A practical target is an active-speech RMS near -20 dBFS with a peak ceiling at -1 dBFS, measured over speech-active frames only, since a file with long pauses otherwise comes out too hot. Apply the gain once. Two normalization passes, one before resampling and one after, produce a corpus whose levels drift in a way that is hard to notice and harder to explain later.

Do not add denoising here by default. Enhancement models introduce artifacts the recognizer has never seen in training, and for moderate noise a recognizer trained on noisy data usually beats a recognizer fed cleaned audio. If the noise is severe enough that a human cannot transcribe the clip, the right fix is more noise in training, not a cleanup step in preprocessing.

Segment, then verify the whole batch

Segment last, after every per-file transform, so that segments are uniform in rate and level. Five to thirty seconds is the useful range for most recognizers: shorter clips waste context, and longer ones make batching awkward.

Then verify the batch against a written list of assertions rather than by ear. Every output file: rate equals the target, one channel, peak at or below -1 dBFS, duration between 0.5 and 30 seconds, RMS above -60 dBFS. Print the count of files that fail each assertion instead of stopping at the first one, because the pattern across the failures is what tells you which step is broken.

The manifest is the deliverable

Keep one row per output file: source path and hash, source rate, output rate, duration before and after trimming, the gain applied, and the version of the script. When someone asks six weeks later why a particular clip sounds wrong, the manifest answers without rerunning anything. It is also what lets you show that a corpus was processed by one pipeline and not by three.