Audio augmentation recipes that help, and the ones that waste compute

Augmentation is not a way to make a small dataset bigger. It is a way to make training conditions resemble the conditions the model will meet in deployment. Framed that way, the recipes worth running are the ones that match a specific gap between your training audio and your deployment audio, and the rest is compute spent for nothing.

Three gaps cover most projects: the microphone and channel, the background noise, and the distance to the speaker. Each has a recipe, and each has a parameter range where it helps and a range where it hurts.

Speed perturbation: the cheapest real gain

Resampling the waveform by a factor changes tempo and pitch together, which is acceptable here because the warp is global and the transcript stays aligned. The standard recipe is three copies at factors 0.9, 1.0, and 1.1. It costs one resample per copy and gives the model duration variation it would otherwise only see from a much wider speaker pool.

Stay inside the range from 0.8 to 1.2. Beyond that, vowel durations and formant spacing move outside the range human speech occupies, and the model starts learning transitions that do not occur in speech. And keep it out of the validation set.

Noise mixing: only with the SNR computed correctly

Mixing noise at a target SNR sounds like one line of code and is usually done wrong in the same way: the noise gain is computed from the RMS of the whole noise file, while the speech level is computed over the speech. If the noise file has quiet passages, the mix lands 5 to 10 dB away from the target.

Compute both levels over speech-active frames, and choose noise segments whose energy is roughly stationary, or tile a segment that is. The useful training band is 5 to 20 dB. Below 5 dB, human intelligibility drops quickly and the model spends capacity on a condition it will rarely meet.

• Matched noise rather than white noise: babble, keyboard, road, and HVAC each teach something different. White noise at any SNR teaches almost nothing about a real room.
• Mix at the same rate as the speech. Resampling noise inside the training loop is a hidden cost paid every epoch.
• Guard the peak. After mixing, if the maximum absolute value exceeds about 0.99, scale the whole mix down rather than clipping it.

RIR convolution: the one that fixes distance

Convolving clean speech with a measured room impulse response is the standard way to simulate far-field audio. Two settings matter. Normalize the impulse response so the sum of its squared samples equals one, which keeps the convolved signal at roughly the original level. Then trim the tail: keeping the first 300 to 500 ms captures the perceptually important part of the reverb and cuts the convolution cost.

A middle ground that is often better than full convolution is early reflections only, meaning the first 50 ms. That simulates the direct path plus the first bounce without the diffuse tail that makes transcripts genuinely ambiguous. For far-field training, pair it with a noise mix at 10 to 20 dB, because rooms are rarely quiet and almost never anechoic.

What usually does not pay off

The failures share one pattern: augmentation that changes the signal more than deployment will.

• Clipping as augmentation. It destroys information the model needs, and real clipping is rarer than the recipes assume.
• Extreme speed or pitch factors, including pitch shifts that break the relationship between formants and the phoneme.
• Augmenting the validation set. Evaluate on unmodified conditions and on separately constructed test conditions, never on the output of the training pipeline.
• Stacking every technique at maximum strength. Masking on top of heavy input augmentation frequently pushes total masking past the point where the training signal survives.

Verify it, and keep it reproducible

Build three evaluation sets: clean, simulated with your augmentation applied offline at a fixed seed, and real recordings made in the actual deployment conditions. Read the three numbers together. If augmentation improves the simulated set and not the real one, the simulation does not match reality, and that is the finding to act on. Watch the clean number as well. A small regression there is normal; a large one means the strength is set for a deployment condition that does not exist.

Apply augmentation in the data loader, per epoch, rather than writing augmented copies to disk once. The model sees a different mixture each epoch, the clean corpus stays clean for evaluation, and storage does not triple. Derive the random seed from the epoch number and the utterance identifier so a run can be reproduced exactly, and record the recipe version next to the seed. Without the seed, an ablation you run next month is not comparable to the one you ran today.