Neural Speech Codec
Short answer. A neural speech codec replaces the waveform with a sequence of discrete tokens from a learned codebook: an encoder maps audio to latent frames, a quantizer snaps each frame to its nearest codebook entry, and a decoder reconstructs audio from the indices. The tradeoff is arithmetic. Bitrate is layers x log2(codebook size) x frames per second, so 1,024 entries at 50 frames per second is 500 bits per second per layer. Bigger codebooks lower the reconstruction error and cost bits, and they need proportionally more training data.
How to do it
What a token is and is not
A codec token is an index into a codebook — an integer, not a phoneme and not a byte of audio. At a 50 Hz frame rate with a 1,024-entry codebook, one second of audio is 50 integers of 10 bits each: 500 bits, against 256,000 bits for the same second as 16-bit 16 kHz PCM. The compression ratio is set by the frame rate and the codebook size, which are design parameters, not properties of the codec family.
The frame rate is the first thing to check when comparing codecs, because it is usually not stated in the headline number. EnCodec at 24 kHz uses 75 Hz frames; SoundStream uses 50 Hz. A 75 Hz codec and a 50 Hz codec at the same codebook size and layer count differ by 50% in bitrate, and the 75 Hz one has finer temporal resolution. Neither is better in general, but a comparison that does not mention the frame rate is comparing two things at once.
The bitrate arithmetic, and what it buys
Bitrate = layers x log2(K) x frame_rate, where K is the codebook size. For a 50 Hz codec with 1,024-entry codebooks: one layer is 500 bps, four layers is 2 kbps, eight layers is 4 kbps. Compare that to the 256 kbps of the source PCM and the compression factor is between 64x and 512x.
The useful operating range for a speech codec is 1.5 to 6 kbps, which is enough to reconstruct intelligible speech and enough for a language model to be trained on the tokens. Below about 1 kbps the reconstruction becomes rough and the tokens carry mostly prosody and coarse spectral shape. Above about 6 kbps you are reconstructing audio quality, which is a different goal from giving a language model a compact representation of the content.
Residual quantization, and why layers help
A single codebook has to cover the whole space of latent frames, and the entries that matter are the ones near the dense regions of the data. Residual vector quantization fixes this by chaining quantizers: layer 1 codes the frame coarsely, layer 2 codes the residual of layer 1, layer 3 codes the residual of layer 2, and so on. Each layer has its own codebook and its own index per frame.
The property this gives you is a nested bitstream. Decoding only the first layer produces intelligible but rough audio; each additional layer refines it. That is why the layer count is a runtime quality knob for a codec like EnCodec, and why a speech language model can be trained on the first one or two layers if it does not need full reconstruction quality. It is also why the layers are not interchangeable: layer 5 codes a small residual and its indices are meaningless without the four before it.
Codebook size is not free
Doubling the codebook from 1,024 to 2,048 entries adds one bit per frame per layer — a 10% bitrate increase at 10 bits — and requires roughly twice as much data to train each entry to the same quality. Entries that are rarely selected stop receiving gradient and drift, which is the codebook collapse failure mode: the effective codebook ends up much smaller than the nominal one.
Monitor codebook utilization, the fraction of entries used by a corpus, and treat it as a first-class metric. Values well below 100% are normal, and a utilization that falls as training proceeds is the signature of collapse. The standard remedies are a commitment loss that pulls the encoder toward the chosen entry, and periodic reinitialization of dead entries to random data points.
Evaluate on the downstream task, not on the reconstruction
Reconstruction error falls monotonically as you add layers, and that is a property of the objective rather than a statement about usefulness. A recognizer consuming the reconstructed audio may not improve, and can get worse: the discrete bottleneck discards phase and fine spectral detail, and the decoder hallucinates plausible detail to fill the gap. The hallucinated content is not in the original audio, and a model trained on it is learning from the decoder as much as from the speech.
So evaluate the codec on what it is for. If it feeds a speech language model, measure the language model's task. If it feeds a recognizer, measure WER on the reconstructed audio. If it is an archive format, measure the storage and the WER. Reconstruction metrics such as mel distance or a perceptual metric are useful for tracking training progress and are not evidence that the codec is good for a downstream task.
Code
Quantize mel frames with a k-means codebook and measure the reconstruction error and codebook utilization as the codebook grows from 8 to 512 entries.
import numpy as np
import librosa
SR, N_MELS, HOP = 16000, 40, 320
rng = np.random.default_rng(0)
def speech_like(dur=12.0, seed=0):
r = np.random.default_rng(seed)
t = np.arange(int(SR * dur)) / SR
f0 = 130 + 25 * np.sin(2 * np.pi * 0.4 * t)
x = sum(np.sin(2 * np.pi * k * f0 * t) / k for k in range(1, 25))
x = x * (0.5 + 0.5 * np.sin(2 * np.pi * 3.0 * t)) ** 2
x = x + 0.01 * r.normal(size=t.size)
return (x / np.max(np.abs(x))).astype(np.float32)
def kmeans(X, K, iters=20, seed=0):
"""K-means with a nearest-centroid assignment step. Stands in for a VQ layer."""
r = np.random.default_rng(seed)
C = X[r.choice(len(X), K, replace=False)].copy()
for _ in range(iters):
idx = np.argmin(((X[:, None, :] - C[None]) ** 2).sum(axis=-1), axis=1)
for k in range(K):
if (idx == k).any():
C[k] = X[idx == k].mean(axis=0)
idx = np.argmin(((X[:, None, :] - C[None]) ** 2).sum(axis=-1), axis=1)
return C, idx
m = librosa.feature.melspectrogram(y=speech_like(), sr=SR, n_mels=N_MELS,
hop_length=HOP)
F = librosa.power_to_db(m, ref=np.max).T # frames x mel bins
F = (F - F.mean(axis=0)) / (F.std(axis=0) + 1e-6)
rate = SR / float(HOP)
print('%d frames at %.0f frames/second, %d mel bins per frame'
% (len(F), rate, N_MELS))
print('for reference, 16-bit 16 kHz PCM is %d bits per second' % (SR * 16))
print()
print('codebook bits/frame bitrate RMS error entries used')
for K in (8, 32, 128, 512):
C, idx = kmeans(F, K)
err = float(np.sqrt(((F - C[idx]) ** 2).mean()))
print('%6d %4d %5.0f bps %.3f %4d / %d'
% (K, int(np.log2(K)), np.log2(K) * rate, err,
len(np.unique(idx)), K))- pip install numpy librosa soundfile. No audio files: the speech is synthesized.
- Expected output: about 600 frames at 50 frames per second. RMS error falls from roughly 0.9 at 8 entries to under 0.2 at 512, and the bitrate column runs from 150 to 450 bps.
- Watch the entries-used column. At K = 512 with only 600 frames, a substantial fraction of the codebook goes unused, which is the codebook-collapse problem from the steps in miniature: there is not enough data to train the entries you allocated.
- Multiply the bitrate by the layer count for a real codec. Four layers at K = 512 is about 1.8 kbps, which is in the range where a speech language model operates.
Where this goes wrong
Comparing codecs without checking the frame rate
Bitrate depends on layers, codebook size, and frame rate, and the frame rate is the one that is usually left out of the headline. Two codecs at 6 kbps with different frame rates have different codebook sizes or different layer counts, and the one with more frames per second is spending its bits on temporal resolution. Ask for all three numbers or you are comparing incomparable designs.
Growing the codebook without growing the data
A codebook entry needs enough examples to be trained. Doubling the codebook halves the examples per entry at the same corpus size, and below a certain ratio the extra entries never become useful: they are assigned nothing, receive no gradient, and the effective codebook is smaller than the nominal one. If utilization is low, the answer is usually more data or a smaller codebook, not a longer training run.
Treating codec tokens as phonemes
A codec token is an index into a learned codebook, and its meaning depends on the codec, the layer, and the training data. Tokens from layer 1 of one codec have nothing to do with tokens from layer 1 of another, and tokens from layer 4 of the same codec only mean something in the context of layers 1 through 3. A pipeline that mixes tokens from two codecs, or that drops the lower layers and keeps the higher ones, produces garbage rather than degraded audio.
Judging the codec by reconstruction metrics
Mel distance and perceptual audio metrics improve monotonically with more layers and more bits, so they cannot tell you where to stop. The stopping point is a downstream decision: the smallest number of layers that keeps the recognizer's WER or the language model's task accuracy where you need it. Optimizing reconstruction beyond that point spends bitrate for nothing.
Ignoring what the decoder invents
A neural decoder reconstructs plausible audio from a discrete bottleneck, which means it fills in detail that was discarded, and the detail it invents is shaped by its training data. If you train a model on reconstructed audio, part of what it learns is the decoder's priors rather than the recording. For most applications that is acceptable and worth the compression; for anything that depends on fine acoustic detail — pronunciation scoring, voice quality, forensic work — it is not, and the original audio is the only honest source.
When to buy the data instead of building the pipeline
Buy when you are training a codec or a model over codec tokens and need audio at a scale and condition diversity that a small corpus cannot provide, because codebook utilization is a data problem and more data is the direct fix. Buy when you need the source material to be clean, since a codec spends its codebook on whatever is in the audio and a corpus with clipping or resampling artifacts teaches it to model the artifacts. Buy when you need coverage of the languages and acoustic conditions the tokenizer will be asked to handle, because a codec trained on one language compresses another less efficiently and the tokens are less useful. If you are testing a quantization scheme or measuring the bitrate-quality curve for a design you are still choosing, the k-means experiment above answers the question and it takes a minute.
More technical reference
-
Word Error Rate (WER)
word error rate
-
Character Error Rate (CER)
character error rate
-
Speaker Diarization
speaker diarization
-
Forced Alignment
forced alignment
-
Voice Activity Detection (VAD)
voice activity detection
-
Diarization Error Rate (DER)
diarization error rate
Need data for Neural Speech Codec?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.