Speech Enhancement
Short answer. Speech enhancement removes noise from a recording. The version you can build in an afternoon is spectral gating: estimate the noise spectrum from a segment where nobody is talking, then attenuate each time-frequency bin by how far it exceeds that estimate, with a floor and a smoothing filter. It works well for stationary noise — fans, hum, hiss — and poorly for competing speech, which is the noise that hurts ASR most. Always report SNR before and after on the same file, computed over speech-active frames only.
How to do it
Get a noise estimate you can defend
Three sources, in descending order of reliability. A known silent segment: a lead-in, a deliberate pause, or a separate recording of the room made at the same session. A percentile over the whole file: the 10th percentile of power per frequency bin, on the assumption that at least a tenth of the file is noise-only. And a noise estimate carried over from a different recording, which is the most common cause of a poor result.
Whichever you use, the estimate is per frequency bin and it needs enough frames to be stable. A per-bin power estimate from 20 frames has roughly a 30% standard deviation, visible in the output as musical noise; 100 frames is comfortable. If your silent segment is shorter than about half a second, smooth across frequency bins as well as time.
Choose the attenuation rule
The classical form is a per-bin gain g = max(1 - alpha x (N/S), floor)^beta, where N is the noise power estimate and S the observed power. Alpha above 1 is over-subtraction, which removes more noise and more speech; beta near 0.5 makes the gain curve gentler. Values that work on stationary noise: alpha = 2.0, beta = 1.0, floor = 10^(-15/20) = 0.18.
The floor is not a safety net, it is a design parameter. It sets the maximum attenuation in dB — a floor of 0.18 is -15 dB — and decides how much of the noise survives. Too high and the noise is still audible; too low and you get isolated surviving bins that warble, which sounds worse than the original noise.
Smooth the gain across frames and frequency
The gain you compute from two noisy power estimates is itself noisy, and it changes from frame to frame in a way the underlying noise does not. That is the mechanism behind musical noise. The cheapest fix is a first-order smoother across frames with a coefficient around 0.5 to 0.9, which costs a fraction of a frame of latency and removes most of the artifact.
Smoothing across three to five adjacent frequency bins helps as well, because the noise power spectrum is smoother than its estimate. Do not smooth the input spectrum itself — smooth the gain, so the phase and the fine structure of the speech are left alone.
Measure SNR correctly, on active frames
Enhancement reduces energy in the noise-only frames more than in the speech frames, so an SNR computed over the whole file is inflated by the improvement in the pauses. Compute the speech power and the residual error power over frames where speech is present — an energy threshold at 10% of the frame maximum — and report the pair. In a simulation with a clean reference the mask is exact; on real audio you need a VAD on the noisy signal, which introduces its own error and is worth stating.
Report the metric on the same file before and after. An enhancement result without the input number cannot be evaluated, and the improvement is easy to overstate because attenuation reduces the speech energy too.
Know the boundary of what spectral gating can do
Stationary broadband noise is the easy case and the method handles it well. Non-stationary noise violates the assumption that the estimate from the silent segment still describes the noise during speech, and a keyboard click or a door will survive with a slight ring around it. Competing speech is the hard case: two talkers occupy the same frequency bins at the same times, and a gain rule that suppresses the interferer also suppresses the target.
Reverberation, clipping, and codec artifacts are not noise in this sense and gating does not address them. If your problem is one of those, the tool is different — dereverberation, or a model trained on the actual degradation. Applying spectral gating to a clipped recording mostly just makes the clipping quieter.
Code
Spectral gating with the noise estimate taken from a speech-free lead-in, reporting SNR over the speech that follows. The measured gain is about 4 dB, not the 20 dB the method is often sold as.
import numpy as np
SR, NFFT, HOP = 16000, 512, 160
WIN = np.hanning(NFFT)
rng = np.random.default_rng(0)
def stft(y):
n = 1 + (len(y) - NFFT) // HOP
return np.stack([np.fft.rfft(y[i * HOP:i * HOP + NFFT] * WIN)
for i in range(n)], axis=1)
def istft(S, length):
out = np.zeros(length)
wsum = np.zeros(length)
for i in range(S.shape[1]):
seg = slice(i * HOP, i * HOP + NFFT)
out[seg] += np.fft.irfft(S[:, i], n=NFFT) * WIN
wsum[seg] += WIN ** 2
return out / np.maximum(wsum, 1e-3)
t = np.arange(int(SR * 4.0)) / SR
f0 = 150 + 10 * np.sin(2 * np.pi * 0.6 * t)
speech = sum(np.sin(2 * np.pi * k * f0 * t) / k for k in range(1, 20))
speech = 0.5 * speech / np.max(np.abs(speech))
speech[:int(SR * 0.5)] = 0.0 # 0.5 s with no speech in it
clean = speech.astype(np.float32)
noisy = (clean + 0.05 * rng.normal(size=t.size)).astype(np.float32)
S = stft(noisy)
P = np.abs(S) ** 2
silent = slice(0, int(0.5 * SR / HOP))
N = P[:, silent].mean(axis=1, keepdims=True) # noise power per bin
gain = np.sqrt(np.maximum(1.0 - 2.0 * N / (P + 1e-12), 10 ** (-15.0 / 10.0)))
for i in range(1, gain.shape[1]): # smooth the gain in time
gain[:, i] = 0.7 * gain[:, i] + 0.3 * gain[:, i - 1]
enhanced = istft(S * gain, len(noisy))
SPEECH_START = int(SR * 0.5) # score past the lead-in
def snr(reference, test):
r = reference[SPEECH_START:]
e = test[SPEECH_START:] - r
return 10.0 * np.log10((r ** 2).sum() / ((e ** 2).sum() + 1e-12))
print('noise estimate from the first 0.5 s, scored over the speech that follows')
print('SNR of the noisy input : %5.1f dB' % snr(clean, noisy))
print('SNR after spectral gating : %5.1f dB' % snr(clean, enhanced))
print('mean gain applied : %5.1f dB' % (20.0 * np.log10(gain.mean() + 1e-12)))
print('energy removed : %.1f%%'
% (100.0 * (1.0 - (enhanced[SPEECH_START:] ** 2).sum()
/ (noisy[SPEECH_START:] ** 2).sum())))- pip install numpy. The STFT and the inverse STFT are written out rather than imported, so the whole method is visible in one file.
- Expected output: 14.0 dB before, 18.1 dB after — about 4 dB. Spectral gating is not a miracle. The ceiling is set by how much of the noise sits in bins the speech does not occupy, and for a harmonic source most of the noise does sit between the harmonics, which is why the number is small.
- The energy-removed line is the honesty check. A small fraction removed for a real SNR gain is the good case — that is what this run shows. A large fraction removed for the same gain means the attenuation is hitting speech, not noise.
- The floor is the parameter people tune first. Lowering it from -15 dB to -30 dB moves this signal barely at all, 18.1 to 18.3 dB, because the floor only binds in bins that are already noise-dominated and carry little energy. On real speech the same change is audible — it is the setting where consonants start to disappear, which is why a metric that improves slightly can sit next to a WER that gets worse.
Where this goes wrong
Estimating the noise from a different session
A noise profile captured yesterday, in another room, or on another device describes a different noise. The estimate is wrong in the bins that matter, the gain is wrong with it, and the output sounds either untouched or hollow depending on which way it missed. If the deployment cannot supply a silent segment per recording, use a per-file percentile instead of a stored profile.
Reporting SNR computed over the whole file
Enhancement suppresses the pauses more than the speech, so a whole-file SNR improves partly because the file got quieter in the parts that were never speech. The number can look like a 15 dB gain when the speech-band improvement is 4 dB. Restrict the measurement to speech-active frames and state how the activity was detected.
Setting the floor to zero for maximum noise removal
A zero floor lets the gain go to zero wherever the estimate exceeds the observation, leaving isolated surviving bins that flicker frame to frame. The artifact is structured noise that no recognizer has seen in training, so it typically costs more WER than the original noise did. A floor between -10 and -20 dB is where most systems land, and the right value is found by evaluating the downstream task, not by listening.
Enhancing the test set but training on unenhanced audio
A model trained on noisy recordings has adapted to that distribution. Feeding it denoised audio at test time is a domain mismatch, and it can be worse than doing nothing — the model is now listening for cues that the processing removed. Either process both sides identically or fine-tune on processed data. This mismatch is one of the most common reasons an enhancement stage shows a metric gain and a WER loss.
Expecting it to remove a competing talker
Two talkers occupy the same bins at the same times, so a per-bin gain rule cannot separate them without removing the target as well. Spectral gating will reduce a babble interferer by a couple of dB at the cost of several dB of target speech. Competing-speech removal is a separation problem and needs a model trained for it, plus data with both talkers labeled.
When to buy the data instead of building the pipeline
Buy when the noise you need to handle is non-stationary or is another voice, because those are the cases a classical method cannot reach and the cases where matched training data decides the outcome. Buy when you need clean and noisy pairs of the same utterances, which is the only way to measure enhancement honestly and the only way to train a supervised model. Buy when the target environment is one you cannot record in — a specific vehicle, a specific headset, a specific room — since the noise spectrum has to match or the estimate is wrong in exactly the bins that matter. If your noise is a fan or a hum and you control the recording, the method above is enough and it costs nothing; building it yourself is the right call for that whole class of problems.
More technical reference
-
Audio Sample Rate
audio sample rate
-
WAV vs MP3 for Training Data
wav vs mp3
-
Opus Audio Codec
opus audio codec
-
Beamforming Microphone Array
beamforming microphone array
-
Neural Speech Codec
neural speech codec
-
Word Error Rate (WER)
word error rate
Need data for Speech Enhancement?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.