SpecAugment

Short answer. SpecAugment is augmentation applied to the log-mel spectrogram instead of the waveform: zero out a few horizontal bands (frequency masks) and vertical bands (time masks) at random before the model sees the batch. It costs nothing at training time — no extra audio, no extra storage, no extra forward passes — and it is one of the few augmentations that reliably helps when transcribed audio is scarce. Mask widths are set as fractions of the input dimensions, masks are drawn per example per batch, and nothing is masked at inference.

How to do it

Start from the published parameter sets instead of guessing

The original paper defines three policies, and they are still the sane starting points. LibriSpeech basic (LB): one frequency mask of up to F = 27 mel bins and one time mask of up to T = 80 frames. LibriSpeech double (LD): two of each, with the widths drawn from a uniform distribution capped at the same F and T.

Put those numbers next to your input shape before you accept them. F = 27 on an 80-bin mel input removes up to 34% of the frequency axis; T = 80 frames at a 10 ms hop is 800 ms, which on a 10-second utterance is 8% of the time axis but on a 1-second utterance is 80%. If your corpus is dominated by short utterances, T = 80 will mask most of the utterance, and the model is being asked to recognize silence. Cap the width as a fraction of the utterance length as well as an absolute frame count.

Draw the masks per example, per batch

The most common implementation bug is drawing one mask set and reusing it for every example in the batch, or worse, for the whole epoch. That reduces the effective augmentation to a single fixed corruption, and it teaches the model to ignore one specific frequency band and one specific time region rather than to be robust to missing information in general. The mask has to be redrawn for every example on every forward pass, which means the augmentation lives in the data pipeline, not in a preprocessing script that writes augmented features to disk.

Pick the fill value deliberately

The paper masks with zero on a log-mel spectrogram that has not been per-bin normalized. If your pipeline normalizes each mel bin to zero mean and unit variance first, then masking with zero is masking with the mean, which is a perfectly plausible frame value — the model cannot tell a masked cell from a quiet one, which is either a feature or a bug depending on what you want.

Masking with a large negative constant makes the mask trivially detectable, and the model learns to route around it rather than to infer through it, which defeats the purpose. Masking with the dataset mean is the middle option and is what most implementations settle on when the features are normalized.

Know what it simulates and what it does not

Time masking simulates dropped frames, clipped audio, or a packet loss — it forces the model to use context on both sides of a gap. Frequency masking simulates a notch filter, a narrowband channel, or a bad microphone response. Both remove information rather than add it, which is why they are cheap and why they are not a substitute for additive noise.

If your deployment has babble or machinery noise, SpecAugment will not teach robustness to it, because the model never sees noise in the masked region — it sees an absence. Mix real noise at real SNRs for that, and keep SpecAugment for the missing-data robustness.

Stack it, then check the validation curve rather than the training loss

SpecAugment operates on features, so it composes with waveform-level augmentation — speed perturbation, pitch shift, volume — which changes the audio before the features are computed. The two together usually beat either alone. The failure mode is over-augmenting: with large masks plus dropout plus aggressive speed perturbation, the training loss goes up and the validation WER stops improving, and the training loss tells you nothing because it is measured on corrupted inputs.

Compare validation WER at a fixed step count, not at the best step count. Augmentation often slows early convergence while raising the final ceiling, so a comparison at 10% of the schedule will favor the unaugmented model for the wrong reason.

Code

Time and frequency masking on a log-mel spectrogram, reporting the shapes, the masked fraction, and the coverage of each axis.

import numpy as np
import librosa

SR, N_MELS, HOP = 16000, 80, 160
rng = np.random.default_rng(3)

def speech_like(dur=10.0, seed=0):
    r = np.random.default_rng(seed)
    t = np.arange(int(SR * dur)) / SR
    f0 = 120 + 15 * np.sin(2 * np.pi * 0.5 * t)
    x = sum(np.sin(2 * np.pi * k * f0 * t) / k for k in range(1, 30))
    x = x * (0.5 + 0.5 * np.sin(2 * np.pi * 3.5 * t)) ** 2
    x = x + 0.002 * r.normal(size=t.size)
    return (x / np.max(np.abs(x))).astype(np.float32)

def logmel(x):
    m = librosa.feature.melspectrogram(y=x, sr=SR, n_mels=N_MELS, hop_length=HOP)
    return librosa.power_to_db(m, ref=np.max)

def spec_augment(spec, n_time=2, n_freq=2, F=27, T=80, fill=0.0):
    """Returns the masked spectrogram and a boolean mask of what was replaced."""
    out = spec.copy()
    mask = np.zeros(spec.shape, dtype=bool)
    n_mels, n_frames = spec.shape
    for _ in range(n_freq):
        width = int(rng.integers(0, F + 1))
        start = int(rng.integers(0, max(1, n_mels - width)))
        out[start:start + width, :] = fill
        mask[start:start + width, :] = True
    for _ in range(n_time):
        width = int(rng.integers(0, T + 1))
        start = int(rng.integers(0, max(1, n_frames - width)))
        out[:, start:start + width] = fill
        mask[:, start:start + width] = True
    return out, mask

spec = logmel(speech_like())
aug, mask = spec_augment(spec)

print('input shape (mel bins x frames): %s' % (spec.shape,))
print('utterance length: %.1f s at a %.0f ms hop'
      % (spec.shape[1] * HOP / SR, 1000.0 * HOP / SR))
print('value range before masking: %.1f to %.1f dB' % (spec.min(), spec.max()))
print('masked cells: %.3f of the spectrogram' % float(mask.mean()))
print('mel bins hit by a frequency mask: %d of %d (%.0f%%)'
      % (int(mask.any(axis=1).sum()), N_MELS, 100.0 * mask.any(axis=1).mean()))
print('frames hit by a time mask: %d of %d (%.0f%%)'
      % (int(mask.any(axis=0).sum()), spec.shape[1],
         100.0 * mask.any(axis=0).mean()))
print('frames fully masked across every mel bin: %d' % int(mask.all(axis=0).sum()))
print('first masked frame index: %d' % int(np.argmax(mask.any(axis=0))))

pip install numpy librosa soundfile. librosa supplies the mel filterbank and the dB conversion.
Expected output: an (80, ~996) input for a 10-second utterance, a masked fraction around 0.5 to 0.7 depending on the widths drawn, and a frequency coverage near 60 to 70% of the mel bins with two masks of up to 27 bins each.
Run it twice: the widths and start positions change every call, which is the behavior you want in a training loop. If your implementation prints the same mask twice, the RNG is being reseeded inside the function.
The last line is a cheap sanity check for the off-by-one that puts a mask at frame 0. A mask that always starts at the first frame is a sign the start index is being computed with the wrong bound.

Where this goes wrong

Applying the mask once per epoch instead of once per batch

This turns SpecAugment into a fixed corruption: the model learns that one particular band is unreliable, which does not generalize to a different band at test time. The augmentation has to run in the batch collation step, drawing fresh widths and positions for every example on every forward pass. Precomputing augmented features to disk is the same mistake with extra storage cost.

Using the paper defaults on short utterances

A time mask of 80 frames is 800 ms at a 10 ms hop. On a 1-second command utterance that is most of the audio, and on a 3-second utterance it is a quarter. The model spends training on inputs that no longer contain the words it is supposed to predict, and WER on short utterances gets worse. Cap the mask width at a fraction of the utterance length — 20% is a reasonable ceiling — in addition to the absolute frame cap.

Leaving SpecAugment enabled at inference

Augmentation is a training-time transform. If the mask is applied in a shared feature function that the evaluation path also calls, test features get corrupted and the reported WER is wrong in a way that looks like a model regression. Guard it behind an explicit training flag, and assert in the evaluation path that no cell equals the fill value.

Masking with a value the model can detect

A large negative constant, or a value outside the range of the real features, is a signal to the model that the cell is artificial. The model then learns to detect the marker rather than to infer the missing content, and the robustness gain disappears while the training loss still looks better. Mask with zero on unnormalized log-mel, or with the per-bin mean on normalized features.

Assuming SpecAugment substitutes for noise augmentation

SpecAugment removes information; noise augmentation adds it. A model trained with only SpecAugment has never seen a competing talker and has no reason to be robust to one, because a masked region is silent rather than noisy. If the deployment environment has noise, mix noise; SpecAugment and noise mixing target different failure modes and both are usually worth having.

When to buy the data instead of building the pipeline

SpecAugment itself is free, so the reason to buy data here is about what augmentation cannot manufacture. Buy when the model needs to handle conditions that no masking pattern resembles — real babble, a specific device, a specific room — because those are additive and the augmentation family is subtractive. Buy when transcribed hours are the bottleneck in the first place, since SpecAugment raises the value of each hour but does not replace the hours. Building it yourself is right when the goal is robustness to clipped, dropped, or partially corrupted audio, which is exactly what this augmentation simulates and where more data will not help.

How buying training data works →

More technical reference

Room Impulse Response

room impulse response
Dereverberation

dereverberation
ASR Evaluation Metrics

asr evaluation metrics
Zero-Crossing Rate

zero crossing rate
Speech Enhancement

speech enhancement
Audio Sample Rate

audio sample rate

All technical reference →

Need data for SpecAugment?

Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.

Pilot batch before the full run, so problems surface early.
Consent documentation delivered with the data.
No medical or clinical data. No recorded telephone calls.