What a single WER number hides about the set behind it
One pooled score can sit on top of a test set where four speakers supply half the words. Here are the four breakdowns that turn the headline into a decision.
An online WER calculator takes two blocks of text and returns one number. The number is arithmetically correct, and that is the whole of what it tells you. Two test sets can produce the same 0.14 while needing completely different engineering work.
The missing information is not in the metric. It is in the test set, and a calculator never sees it. Every breakdown below is a group-by on a manifest you either have or need to ask for.
Speaker is the first cut, because it is the widest
Pool the edits and the reference words per speaker, then divide. Do not average per-utterance WER values; sum the edits and sum the words, because that reproduces the pooled calculation and keeps the long files weighted correctly. Print the word share next to each row.
The row that matters is rarely the mean. If two speakers hold 40% of the reference words and their combined WER is 0.08 while the other twenty sit at 0.19, the headline is a description of those two people. That is a test set problem, and no amount of model work will move the number until the set is rebalanced.
The spread across speakers is also a stability signal. A minimum of 0.05 and a maximum of 0.44 on the same model means the score will swing depending on who is in the next recording session. Report the minimum and the maximum, not just the mean.
Duration bands catch the failure the average smooths over
Bucket the set into under 3 seconds, 3 to 10 seconds, 10 to 30 seconds, and over 30 seconds, and compute the pooled WER inside each bucket. Errors concentrate in the shortest bucket, because a short clip carries no context, the first and last words are often clipped, and a single error moves a small denominator.
The pooled number is then dominated by whichever band holds the most words, which is usually the long band. If the product being evaluated is a voice command interface that receives two-second utterances, a test set of thirty-second recordings reports a number that will not transfer. Report the per-band rows and state which band matches production.
Split the error into substitutions, deletions and insertions
jiwer.process_words returns the three counts separately on the output object, and the mix is a diagnosis. Report each count as a share of reference words so the three add up to the WER and the reader can see the composition.
A high deletion share means the system is dropping audio: usually a voice activity threshold set too aggressively, a maximum output length, or truncated files in the harness. A high insertion share means it is inventing words over silence or noise, which is the failure mode that looks harmless in a transcript and is fatal in a search index.
Two systems at 0.14 are not comparable products when one is 12% deletions and the other is 12% insertions. The first is losing speech, the second is hallucinating it.
Recompute the headline under production weights
Once you have per-slice error rates, you can reweight. Take the edit and word counts per slice, apply weights that match the traffic mix you expect in production, and recompute. If the headline moves by more than the interval around it, the number was a property of the test set rather than of the model.
Print both: the set as collected, and the set reweighted to production. When they disagree, the second one is the estimate and the first one is a historical artifact of how the data was gathered. Say which weights you used, because a weighted number with hidden weights is a worse artifact than an unweighted one.
The manifest columns are the real deliverable
Every breakdown above is a group-by, which means the set has to carry the columns. Speaker identifier, duration in seconds, device or channel, sample rate, and language. A test set delivered as audio plus transcripts can only ever produce the single number.
That is why the columns belong in the specification rather than in a follow-up email. If you are buying evaluation data, ask for the manifest before you ask for the audio, and check that a speaker identifier exists and is stable across files. Everything in this article is one afternoon of work given those columns, and impossible without them.