Technical reference

The metrics, formats and pipeline steps that decide whether a speech dataset is usable. Each page opens with the answer, then the steps, then working code, then the mistakes that actually cost projects time.

Word Error Rate (WER)

WER is the number of word-level edits needed to turn the system output into the reference, divided by the number of words in the reference: WER = (S + D + I) / N. A score of 0.15 means 15 edits per 100 reference words. It can exceed 1.0, because insertions are counted against a denominator that does not grow. Lower is better, and there is no universal passing grade — a read audiobook and a noisy call center tolerate very different numbers.
Character Error Rate (CER)

CER is the same edit-distance calculation as WER with characters as the unit instead of words: CER = (S + D + I) / N, where N is the number of characters in the reference. It is the correct metric for languages written without spaces — Japanese, Thai, Chinese — where a word-level score has no boundaries to count. CER and WER are not convertible, because the tokenization that would convert them is the thing that was broken.
Speaker Diarization

Speaker diarization answers "who spoke when": it segments audio into speaker turns and labels each turn. The labels are arbitrary, so a diarization system is scored on two separate things — whether the turn boundaries are in the right place, and whether the turns group correctly by speaker. Comparing labels directly gives 0% agreement on a perfect system. Scoring requires mapping hypothesis labels onto reference labels first, usually by majority overlap.
Forced Alignment

Forced alignment takes audio and its transcript — both already known — and finds the start and end time of every word or phoneme. It is not recognition: the text is an input, not an output. Production systems score every frame against every phoneme with an acoustic model, then run a Viterbi search for the best path through the known text. A simple energy-based segmentation works only when words are separated by silence, which makes it a demonstration rather than a solution.
Voice Activity Detection (VAD)

VAD labels each frame of audio as speech or non-speech, usually every 10 to 30 ms. The simplest usable version combines two features: frame energy, which says something is there, and zero-crossing rate, which says whether it is tonal like voiced speech or broadband like noise. Energy alone fails on any loud non-speech sound. Add hysteresis — separate thresholds to enter and leave a speech run — or the output chatters on and off at every frame boundary.
Diarization Error Rate (DER)

DER is the fraction of reference speech time that is wrong, broken into three components: missed speech (reference speech the system called silence), false alarm speech (non-speech the system called speech), and speaker confusion (speech attributed to the wrong speaker). DER = (missed + false alarm + confusion) / total reference speech time. The standard convention ignores 0.25 s around every reference speaker change, because human annotators cannot place boundaries more precisely than that.
Sample Rate for Speech Recognition

16 kHz is the working standard for speech recognition, because it captures everything up to 8 kHz and human speech carries almost no useful energy above that. The Nyquist limit says a sample rate captures frequencies up to half the rate: 16 kHz reaches 8 kHz, 8 kHz reaches 4 kHz. Recording below the needed rate does not just lose high frequencies — it folds them back down as phantom tones that were never in the original. That is aliasing, and it is permanent.
MFCC vs Mel Spectrogram

A mel spectrogram is a spectrogram with mel-spaced frequency bands; a log-mel spectrogram is the same thing in decibels. MFCCs are a log-mel spectrogram with a discrete cosine transform applied and all but the first 13 to 20 coefficients thrown away. The transform decorrelates the bands and the truncation compresses the data, but the truncation is lossy: keeping 13 of 80 coefficients changes the representation by about 3 dB per band on average.
Audio Augmentation

The three augmentations that pay off for speech recognition are additive noise mixed at a target SNR, small time shifts, and speed perturbation. All three preserve the transcript, which is what makes them cheap: no re-annotation is needed. The catch is that speed perturbation changes the duration and time shifts move the start offset, so the manifest has to be rewritten even though the text does not change. Augment and update the manifest in the same pass, not as an afterthought.
Mean Opinion Score (MOS)

MOS is the mean of listener ratings on a 1-to-5 scale, and on its own it is not evidence of anything. A MOS of 4.0 from 8 listeners carries a 95% confidence interval roughly 1.5 points wide, and repeated 8-listener studies of the same system land about 1.0 point apart. From 40 listeners that spread narrows to 0.45. Report the listener count and the interval next to the number, or it is not comparable to anything.
Far-Field Speech Recognition

Far-field recognition fails for two reasons that need different fixes. Distance costs level: sound drops 6 dB every time the distance doubles, so moving a microphone from 0.3 m to 4 m loses 22.5 dB of SNR. Reverberation smears time: reflections arrive after the direct sound and overlap the next phoneme. Level loss can be partly recovered with gain. The smearing cannot, because it is a change to the signal rather than to its volume.
PII Redaction in Speech Transcripts

PII redaction over a transcript is a span problem: find every match, resolve the overlaps, then replace. The order matters, because a phone pattern and a longer numeric-ID pattern will match the same digits and the shorter one must not win. Masking style matters too: replacing a span with a same-length mask keeps every character offset intact, so an alignment computed before redaction stays valid. A bracketed label such as [EMAIL] reads better but shifts every offset after it.
Multilingual Speech Recognition

A pooled multilingual WER weights every word equally regardless of language, so a language with ten times the word count dominates the headline. In a five-language set where English holds 59% of the reference words, a language failing at 65% WER moves the pooled number by less than one point. Report per-language WER next to the pooled figure, and run a leave-one-language-out pass to see which language is actually setting it.
Signal-to-Noise Ratio (SNR)

SNR is the ratio of speech power to noise power in decibels: SNR = 10 log10(P_speech / P_noise). The number depends entirely on where you measured it. Computed over a whole file that contains pauses, the speech power is diluted by the silent frames, so the same recording reports 10.0 dB over the full file and 15.8 dB over the speech frames only. Always state the measurement window with the number.
Mel Spectrogram

A mel spectrogram is a spectrogram whose frequency axis has been warped onto the mel scale, which spaces bands by perceived pitch rather than by hertz. The mapping is mel = 2595 log10(1 + f / 700), so bands are narrow at low frequency and wide at high frequency — about 44 Hz near zero and about 519 Hz at the top of a 40-band, 16 kHz configuration. The warp puts resolution where speech information is.
Transcription Accuracy

Transcription accuracy is measured on a held-out sample, not on the whole corpus, and the sample has to be stratified by the conditions you care about — SNR, speaker, duration, accent. A proportion carries a confidence interval: on 100 clips it is about 11 points wide, and on 1,600 clips about 2.5. Measuring on the portion used for tuning inflates the number, by about 3 points in the example below.
Speaker Verification

Speaker verification decides whether two recordings are the same person. The standard approach turns each recording into a fixed-length embedding and compares them by cosine similarity, accepting the pair when the score passes a threshold. The two error rates trade off: raising the threshold rejects more impostors and more genuine speakers. The threshold where the two rates are equal is the equal error rate, and it is the usual single-number summary.
Speaker Identification vs Speaker Verification

Identification asks which of your enrolled speakers this is: a closed-set N-way choice, scored by top-1 accuracy, where the error rate grows as you add speakers. Verification asks whether this is the claimed speaker: a yes/no decision against a threshold, scored by false accept and false reject rates, where the error rate depends on where you set that threshold. The same embedding model serves both tasks, but the numbers are not interchangeable, and a verification score is not a probability.
Speaker Embedding

A speaker embedding is a fixed-length vector — typically 128 to 512 floats — that represents the voice in an utterance, so two utterances can be compared with one cosine similarity. The classical recipe pools statistics over frames (mean and standard deviation of MFCCs, concatenated); the modern one comes from a trained network (x-vector at 512 dimensions, ECAPA-TDNN at 192) and separates speakers far better at fewer dimensions. The vector is not interpretable dimension by dimension, and it carries channel and phonetic content along with identity.
Phoneme Recognition

Phoneme recognition converts audio into a sequence of phoneme symbols — usually the 39-phone ARPAbet set for English — and it is scored with phoneme error rate (PER), the edit distance between the reference and hypothesis phone strings divided by the reference phone count. PER is not WER on a smaller alphabet: English speech carries roughly 3 to 5 phones per word, so one word error usually costs several phone errors, and a single phone substitution costs one ninth of a word. Stress marks are a second axis of error and must be scored by a stated convention.
Code-Switching Speech Recognition

Code-switching ASR transcribes speech that alternates between two languages mid-sentence, and it is scored against language-tagged tokens so the reference records which language each word was in as well as what was said. The failure mode unique to this task is normalization: if the reference writes a borrowed word in Devanagari and the system outputs the Latin spelling, an unnormalized WER counts every character as an error and the score approaches 1.0 on a transcript that is substantially correct. Fix the script policy, the tag convention, and the loanword spelling before you train or evaluate anything.
Accented Speech Recognition

Accented speech recognition is the same modeling problem as any other ASR, but the number that matters is not the pooled WER — it is the spread across accent groups. A system at 12% pooled WER can sit at 4% on the majority accent and 38% on the worst group, and the pooled figure hides that because the majority group contributes most of the words. Always report per-group WER with the reference word count for each group, and check whether the test set accent mix resembles the deployment population before you quote anything.
Noise-Robust Speech Recognition

Noise-robust ASR is measured with a WER-versus-SNR curve, not a single WER number. Mix the same clean test set with the same noise at a ladder of signal-to-noise ratios — 20, 15, 10, 5, 0, -5 dB — and report the whole curve plus the noise type, because a system that wins at 10 dB in white noise can lose at 10 dB in babble. One SNR number is a point on a curve your deployment will not sit on, and the shape of the curve near your operating point is the only part that transfers.
SpecAugment

SpecAugment is augmentation applied to the log-mel spectrogram instead of the waveform: zero out a few horizontal bands (frequency masks) and vertical bands (time masks) at random before the model sees the batch. It costs nothing at training time — no extra audio, no extra storage, no extra forward passes — and it is one of the few augmentations that reliably helps when transcribed audio is scarce. Mask widths are set as fractions of the input dimensions, masks are drawn per example per batch, and nothing is masked at inference.
Room Impulse Response

A room impulse response (RIR) is the recording of what a room does to a click: a direct arrival, then reflections that decay over time. Convolve clean speech with an RIR and you get that speech as it would sound in that room, which is how reverberant training data is made from clean data. The number that summarizes an RIR is RT60, the time for the reverberant energy to fall 60 dB. The number that predicts ASR difficulty is the direct-to-reverberant ratio, because a long RT60 with a strong direct path is easier than a short one without.
Dereverberation

Dereverberation removes the late reverberant tail from a signal. It cannot undo the smearing the early reflections already applied, because those reflections overlap the direct sound and there is no way to separate them after the fact. The workhorse method is spectral subtraction against an estimate of the late-reverberation power spectrum, obtained by exponentially smoothing past frames of the observed spectrum. It runs per channel before the ASR front end, and the only honest metric is a before-and-after comparison on the same signal.
ASR Evaluation Metrics

Word error rate is edit distance over words divided by the reference word count; character error rate is the same at character level; sentence error rate counts an utterance wrong if it contains any error at all. They are one formula with three units, and they do not rank systems the same way. WER under-weights short utterances because long ones dominate the denominator; SER saturates near 100% on long-form audio; CER tracks WER when errors are whole words and diverges when they are partial. Pick the metric that matches what a failure costs your product.
Zero-Crossing Rate

Zero-crossing rate is the number of times the waveform changes sign per frame, reported either as a count or normalized by frame length. It costs almost nothing to compute and it separates fricatives from vowels in clean audio: /s/ and /f/ run 10 to 20 times higher than /a/ or /i/, because a voiced vowel crosses zero about twice per pitch period while a fricative is noise-like. What it cannot tell you is whether a signal is voiced — a silent frame with a high-frequency noise floor has a higher ZCR than any vowel.
Speech Enhancement

Speech enhancement removes noise from a recording. The version you can build in an afternoon is spectral gating: estimate the noise spectrum from a segment where nobody is talking, then attenuate each time-frequency bin by how far it exceeds that estimate, with a floor and a smoothing filter. It works well for stationary noise — fans, hum, hiss — and poorly for competing speech, which is the noise that hurts ASR most. Always report SNR before and after on the same file, computed over speech-active frames only.
Audio Sample Rate

Sample rate is the number of amplitude measurements per second, and it sets the highest frequency a recording can represent: half the sample rate, called the Nyquist limit. Speech is intelligible with energy up to about 4 kHz, which is why the telephone band is 8 kHz, but fricatives carry energy up to 8 to 10 kHz, which is why 16 kHz is the default for ASR training data. Downsampling without a low-pass filter first folds high frequencies back into the audible band as aliasing, and no later processing can undo it.
WAV vs MP3 for Training Data

WAV stores uncompressed PCM: 16-bit mono at 16 kHz is 32 kB per second, 115 MB per hour, 115 GB for 1,000 hours. MP3 at 128 kbps is 58 MB per hour, half the size, and at 32 kbps it is 14 MB per hour, an eighth, at a quality cost that shows up as a WER increase in the model you train on it. For training data the rule is simple: archive in WAV or FLAC, ship lossless, and use a lossy format only when the storage saving is worth an accuracy loss you have actually measured.
Opus Audio Codec

Opus is the codec behind WebRTC, most real-time voice, and a growing share of archived speech. Two properties break naive decoding. It always works internally at 48 kHz regardless of the input rate, so a 16 kHz source decodes to 48 kHz unless you ask otherwise. And the decoder emits a fixed pre-skip of 312 samples — 6.5 ms at 48 kHz — of lead-in that has to be trimmed, or every timestamp in your pipeline shifts by that amount. Frames are 2.5 to 60 ms, with 20 ms as the default.
Beamforming Microphone Array

A beamformer combines the channels of a microphone array with per-channel delays so that sound from one direction adds up coherently and sound from other directions does not. Delay-and-sum with M microphones gives up to 10 log10(M) dB of gain against uncorrelated noise — 6 dB for four microphones, 9 dB for eight — but only for a source in the steering direction, and only while the spacing stays under half a wavelength. Above that spacing the array has grating lobes and picks up directions you did not intend.
Neural Speech Codec

A neural speech codec replaces the waveform with a sequence of discrete tokens from a learned codebook: an encoder maps audio to latent frames, a quantizer snaps each frame to its nearest codebook entry, and a decoder reconstructs audio from the indices. The tradeoff is arithmetic. Bitrate is layers x log2(codebook size) x frames per second, so 1,024 entries at 50 frames per second is 500 bits per second per layer. Bigger codebooks lower the reconstruction error and cost bits, and they need proportionally more training data.

Why this section exists

Almost every rejected dataset we see was rejected for a reason that was visible at specification time. A metric was quoted without saying what it was measured on. A sample rate was chosen by habit. A transcript convention was never written down, so two annotators produced two different texts from the same recording.

These pages are the reference we would hand a buyer before a project starts. They are written for the engineer who has to make the decision, not for a glossary.

Looking for term definitions instead? →

Have a specification you want checked?

Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.

Pilot batch before the full run, so problems surface early.
Consent documentation delivered with the data.
No medical or clinical data. No recorded telephone calls.