Mel Spectrogram
Short answer. A mel spectrogram is a spectrogram whose frequency axis has been warped onto the mel scale, which spaces bands by perceived pitch rather than by hertz. The mapping is mel = 2595 log10(1 + f / 700), so bands are narrow at low frequency and wide at high frequency — about 44 Hz near zero and about 519 Hz at the top of a 40-band, 16 kHz configuration. The warp puts resolution where speech information is.
How to do it
What the warp does to the frequency axis
A linear spectrogram divides 0 to 8000 Hz into equal-width bins. At 16 kHz with a 512-point FFT every bin is 31.2 Hz wide, whether it sits at 100 Hz or at 7000 Hz.
That is a poor match for hearing. Pitch perception is roughly logarithmic above 1 kHz, and speech information is concentrated at low frequencies. The mel scale approximates that perceptual spacing, and warping the axis onto it gives you more bands where the information is and fewer where it is not.
The formula, and what it implies
The common form is mel = 2595 log10(1 + f / 700), with the inverse f = 700 (10^(m/2595) - 1). The 700 constant sets the corner where the scale starts to bend: below roughly 700 Hz the mapping is close to linear, above it the bands widen steadily.
Build n_mels + 2 evenly spaced points on the mel axis, convert them back to hertz, and you have the band edges. In a 40-band configuration at 16 kHz the first band is 44.4 Hz wide and the last is 518.6 Hz wide — a factor of twelve between the bottom and the top of the same filterbank.
The filterbank is fixed, not learned
Each mel band is a triangular filter over the linear FFT bins, and the whole filterbank is computed once from the sample rate, the FFT size and the band count. Nothing about it is trained. That is why the mel spectrogram is cheap, and why changing the band count changes the representation rather than the model.
Typical choices: 40 bands for older speech systems, 80 for modern ASR, 128 for general audio and music models. More bands means more frequency detail and a larger input; for 16 kHz speech, 80 is the current default.
Reading the output
The raw mel spectrogram is a power value per band per frame, and those values span several orders of magnitude, so it is almost always converted to decibels before use. librosa.power_to_db with a reference does that, and the reference choice matters: ref=np.max makes 0 dB the loudest point in the file, which removes absolute level.
If absolute level matters — and for anything involving SNR or gain it does — use a fixed reference instead, or the same recording at two volumes will produce identical features.
Code
Build a 40-band mel filterbank by hand from the Hz-to-mel formula, apply it to a three-tone signal, and see how wide the bands get toward the top.
import numpy as np
import librosa
SR = 16000
t = np.arange(SR) / SR
sig = (np.sin(2 * np.pi * 440 * t) + 0.5 * np.sin(2 * np.pi * 1320 * t)
+ 0.25 * np.sin(2 * np.pi * 3520 * t))
sig += 0.005 * np.random.default_rng(0).standard_normal(len(sig))
n_fft, hop, n_mels = 512, 160, 40
S = np.abs(librosa.stft(sig, n_fft=n_fft, hop_length=hop)) ** 2
fb = librosa.filters.mel(sr=SR, n_fft=n_fft, n_mels=n_mels)
mel = fb @ S
logmel = librosa.power_to_db(mel, ref=np.max)
print(f"linear spectrogram {S.shape} bin width {SR / n_fft:.1f} Hz at every frequency")
print(f"mel filterbank {fb.shape} {n_mels} bands, non-uniform")
print(f"log-mel {logmel.shape}")
def hz_to_mel(f):
return 2595.0 * np.log10(1.0 + f / 700.0)
def mel_to_hz(m):
return 700.0 * (10 ** (m / 2595.0) - 1.0)
edges = mel_to_hz(np.linspace(hz_to_mel(0.0), hz_to_mel(SR / 2), n_mels + 2))
widths = np.diff(edges)
print(f"\nfirst six band edges (Hz): {np.round(edges[:6], 1)}")
print(f"last five band edges (Hz): {np.round(edges[-5:], 1)}")
print(f"band width near 0 Hz {widths[0]:>7.1f} Hz")
print(f"band width near 4 kHz {widths[np.argmin(np.abs(edges[:-1] - 4000))]:>7.1f} Hz")
print(f"band width at the top {widths[-1]:>7.1f} Hz")
energy = mel.mean(axis=1)
print("\nbands lit up by the three tones (440, 1320, 3520 Hz):")
for b in np.flatnonzero(energy > 0.01 * energy.max()):
print(f" band {b:>2} {edges[b]:>7.1f} - {edges[b + 1]:>7.1f} Hz "
f"{10 * np.log10(energy[b] / energy.max()):>6.1f} dB")- pip install numpy librosa
- The band edges are built from the formula by hand rather than taken from the library, so you can see the warp: 0, 44.4, 91.6, 141.7, 195.1, 251.8 Hz at the bottom and 6103.7 up to 8000 Hz at the top.
- Band width goes from 44.4 Hz at the bottom to 518.6 Hz at the top. The 3520 Hz tone lands in band 29 (3461.0 to 3724.8 Hz) at -17.6 dB, which is why the threshold is 1% of peak energy rather than something higher.
Where this goes wrong
Normalizing to the file maximum
ref=np.max makes the loudest frame 0 dB in every file, so a quiet recording and a loud one produce the same features. That is usually what you want for recognition and never what you want if level is a feature or if you are measuring SNR.
Changing n_mels without changing the model input
The band count is baked into the input layer width. A model trained on 80 bands will not accept 40, and a fine-tune that changes the band count needs the input layer replaced. Record the band count alongside the dataset.
Forgetting that the FFT size sets the low-frequency resolution
The mel filterbank cannot create resolution the STFT does not have. With a 512-point FFT at 16 kHz the first mel bands are narrower than the 31.2 Hz bin spacing, so several of them sample the same bins and four of your input rows carry one row of information. Use a larger FFT when the low bands matter.
Assuming two mel spectrograms of the same shape are the same features
Filterbank normalization (area versus peak), the power exponent, and the dB floor all differ between libraries and versions. Two 80-band mel spectrograms can be systematically offset from each other by several dB, which is enough to break a model at inference while every shape check still passes.
When to buy the data instead of building the pipeline
Build the feature extractor — it is one library call and its parameters should match your model rather than your data supplier. Buy when the audio conditions are the variable and you need them labeled: a corpus with documented sample rate, microphone and environment across every file, because a mel spectrogram computed over inconsistent audio encodes the inconsistency. If your files are a mix of sample rates and channel counts, the front end will produce a representation your model cannot learn from.
More technical reference
-
Transcription Accuracy
transcription accuracy
-
Speaker Verification
speaker verification
-
Speaker Identification vs Speaker Verification
speaker identification vs speaker verification
-
Speaker Embedding
speaker embedding
-
Phoneme Recognition
phoneme recognition
-
Code-Switching Speech Recognition
code switching speech recognition
Need data for Mel Spectrogram?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.