Linguacorpus

Word Error Rate (WER)

Short answer. WER is the number of word-level edits needed to turn the system output into the reference, divided by the number of words in the reference: WER = (S + D + I) / N. A score of 0.15 means 15 edits per 100 reference words. It can exceed 1.0, because insertions are counted against a denominator that does not grow. Lower is better, and there is no universal passing grade — a read audiobook and a noisy call center tolerate very different numbers.

How to do it

Score first, then open up the arithmetic

A word error rate is not an accuracy. It is a distance: the smallest number of edits that turns the hypothesis into the reference, counted in words. Three kinds of edit exist — a substitution (the wrong word), a deletion (a word the system missed), and an insertion (a word the system added). The reference length N is the denominator, and it is fixed before scoring starts.

The code below prints both halves of that: the single number you would put in a report, and the S, D, I counts that produced it. Always look at the counts. A WER of 0.30 made of deletions means the system is dropping audio; the same 0.30 made of insertions means it is inventing words through noise. Those are different bugs with different fixes.

Why the score can pass 1.0

Because the denominator is the reference length and the numerator is not bounded by it. If a system emits a hundred words over a ten-word reference, I is 90 and WER is 9.0. This surprises anyone who reads WER as a percentage, and it is the reason you should never clamp a reported WER to 1.0 in a dashboard.

A system that fails this way is usually one that has not been given a speech/non-speech decision, so it transcribes the noise floor and produces a stream of insertions over the silence.

Pool the counts, do not average the scores

The corpus WER is the sum of all edits divided by the sum of all reference words. It is not the mean of the per-file WERs. The two diverge sharply when file lengths differ: a five-word clip with one error scores 0.20 and a five-hundred-word file with twenty errors scores 0.04, but as files they count equally in a naive average while the second holds four times as much evidence.

  • Pooled: (all edits) / (all reference words). This is the number to report.
  • Macro-averaged: mean of per-file WER. Higher than pooled on almost every real set, because short files dominate.
  • If you need a macro number, weight it by reference length — which reproduces the pooled calculation.

Normalize the same way on both sides

Every WER number is only meaningful next to the normalization rules that produced it. Case folding, punctuation stripping, number expansion, and filler-word removal each move the score by more than most model changes do.

Pick a normalization, write it down, and apply it identically to the reference and the hypothesis. When you compare two vendors, compare their normalization first — a five-point WER gap between two systems is routinely a punctuation-handling difference rather than an acoustic one.

Code

Score two pairs with jiwer, then rebuild the same number by hand from the substitution, deletion and insertion counts.

from jiwer import process_words

PAIRS = [
    ("the quick brown fox jumps over the lazy dog",
     "the quick brown box jumped over a lazy dog"),
    ("please record the sentence",
     "please record the whole sentence"),
]

for ref, hyp in PAIRS:
    out = process_words(ref, hyp)
    n = len(ref.split())
    s, d, i = out.substitutions, out.deletions, out.insertions
    print(f"ref: {ref}")
    print(f"hyp: {hyp}")
    print(f"  hits={out.hits} subs={s} dels={d} ins={i}  reference words={n}")
    print(f"  WER = (S + D + I) / N = ({s} + {d} + {i}) / {n} = {(s + d + i) / n:.3f}")
    print(f"  jiwer reports {out.wer:.3f}\n")
  • pip install jiwer
  • The first pair is three substitutions over nine reference words, so WER = 3/9 = 0.333. The second is one insertion over four words, so WER = 1/4 = 0.250. jiwer agrees with both hand calculations.
  • process_words returns the alignment, so out.substitutions is a count, not a list. Use out.alignments if you need to see which words were matched.

Where this goes wrong

Stripping punctuation from the reference but not the hypothesis

If the model emits punctuation and your reference has none, every comma and period becomes an insertion. On a hundred-word reference with twenty marks, that is a 0.20 WER manufactured entirely by the preprocessing step. Strip both sides or neither, and say which in the report.

Averaging per-file WER

The mean of per-file WERs is not the corpus WER, and it is almost always higher, because a three-word file and a three-hundred-word file carry the same weight. Report the pooled number and label it as pooled.

Comparing your number to a published one

A WER quoted from a paper was computed under that paper's normalization on that paper's test set. Unless both match yours the comparison is noise. A model at 0.05 on read speech is not better than yours at 0.12 on call center audio.

Reporting one WER for a multi-condition set

A pooled WER over clean and noisy audio hides which half failed. A corpus that is 80% clean and 20% noisy can report 0.08 overall while the noisy fifth sits at 0.30. Split by condition, by speaker, and by accent before quoting the total, or you will ship a model that is excellent on the easy half and broken on the half your users are in.

When to buy the data instead of building the pipeline

Build the scorer yourself — it is thirty lines and you should own it so you can change the normalization. Buy the data when you need a number that means something: more than a few hundred hours of evaluated audio, speakers outside the accents you can recruit locally, or a language where no public test set exists. A test set drawn from your own production traffic is the most honest benchmark there is, and it is also the one nobody can hand you. We assemble evaluation sets with the same speaker and condition breakdown as the training data, kept separate, so the number you report is the number you will see.

How buying training data works →

More technical reference

All technical reference →

Need data for Word Error Rate (WER)?

Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.

  • Pilot batch before the full run, so problems surface early.
  • Consent documentation delivered with the data.
  • No medical or clinical data. No recorded telephone calls.

We reply within two business days. Your details are used only to answer this request. See our privacy policy.

Contact

Talk to a human

Send a specification and we will come back with a real number and timeline.

Submit a sourcing request

Or email hello@linguacorpus.com