Choosing a masking policy for speech augmentation

Masking is the cheapest augmentation available. It operates on features rather than on audio, so it costs no extra storage, no extra recordings, and no extra forward passes. It is also the augmentation most often left at a default that was tuned on someone else corpus.

The policy question is not whether to mask. It is how much to mask, along which axis, and scaled to what, and those three answers change with the model and with the language.

Set a budget, not a mask width

The parameters people usually set are the number of masks and the width of each one. The quantity that actually determines whether training works is the total share of the input that gets replaced, per example, and that is a product: the number of masks times the mean width, divided by the size of the axis. Two settings that look different can produce the same budget, and two settings that look the same can produce very different budgets on different corpora.

Compute the realized value on your own data before trusting any published setting, because the same absolute width covers a very different share depending on the utterance. A mask that removes a small share of a ten-second utterance removes most of a one-second utterance, and a model that receives it is being asked to recognize a gap with no context on either side.

The rule that follows: express the width as a share of the utterance or of the axis, with an absolute cap, rather than as a fixed number of frames. Then look at the distribution of the realized masked share across the corpus, not just the mean, because the interesting failures live in the tail.

The two axes fail differently

Time masking removes frames, which forces the model to infer a missing span from the context on both sides. It models dropped packets, clipped audio, and short dropouts - a missing-data problem. Frequency masking removes bands, which forces the model to identify a sound from partial spectral evidence, and it models a microphone with a poor response, a narrowband channel, or a notch filter.

Both are legitimate, and they teach robustness to different deployment failures. Masking the wrong axis rarely costs much accuracy, because the model simply learns to be robust to something it will not encounter. The exception is when the masked axis carries the information the task depends on, which is the next problem.

Where frequency masking is the wrong tool

In a tonal language, the tone of a syllable is carried by the fundamental frequency and the spacing of its harmonics. A frequency mask removes a band of mel bins, and that band contains part of the tone pattern. With a large corpus the model recovers the tone from context. With a few hours it learns to guess, and guessing tone is exactly the error that minimal pairs punish hardest. If the corpus is small and the language is tonal, keep frequency masks narrow or drop them and rely on time masking plus waveform-level augmentation.

Two related cases are worth knowing. Narrowband audio: on telephone-band recordings the informative spectrum is a narrow region, so a fixed number of mel bins removes a much larger share of the useful information than the same number would at wideband. Fricatives: they are broadband and low in energy, so a frequency mask can remove the only evidence that distinguishes one from another, and the error it produces is a substitution that no language model will repair.

Scale the mask to the utterance, not to the tensor

Models that pad to a fixed window behave differently from models that pad to the longest item in the batch. A model that pads to the batch maximum sees a mask of a given length as a smaller share of a long utterance than of a short one, which is imperfect but at least proportional to something real. A model with a fixed thirty-second input window is a different story: a three-second utterance occupies a tenth of that input, so a mask drawn uniformly over the tensor lands in padding most of the time. The realized masked share of actual speech ends up a fraction of the configured value, and the augmentation is much weaker than the settings suggest.

The fix is to draw mask positions inside the real audio region, using the attention mask or the known duration, and to compute the mask over the unpadded features before padding. It is a small change to the augmentation code, and it is the difference between the policy you configured and the policy you are actually running.

Schedules, and the settings that look like details

A fixed policy for the whole run is the default and is usually fine. Annealing, meaning wide masks early and narrow masks late, appears in some recipes on the theory that the model first learns robustness and later learns precision. The evidence for annealing is weaker than the evidence for getting the budget right, and it adds a schedule to tune, so it is the wrong first thing to reach for. The details that matter more:

• Draw masks per example per step. A mask shared across a batch teaches the model to ignore one specific region, which is a worse outcome than no augmentation at all.
• Seed the mask generator per process in distributed training. If every worker draws from the same seed, all of them apply identical masks to their own batches and the effective augmentation is divided by the number of workers.
• Keep masking out of validation and inference. Some implementations leave the augmentation path enabled inside the model rather than in the data pipeline, and a model that masks during evaluation reports a number it does not deliver.
• Do not stack two masking mechanisms by accident. If feature masking is already on by default in the fine-tuning configuration and you also apply your own, the realized budget is the sum rather than the maximum.
• Build the test set that justifies the policy: recordings that already contain dropouts, plus clean audio with short segments deleted. A correctly set policy moves the general test set slightly and the dropout set a lot. If it does not move the dropout set, the masks are not reaching the audio that matters.