Linguacorpus

ASR Evaluation Metrics

Short answer. Word error rate is edit distance over words divided by the reference word count; character error rate is the same at character level; sentence error rate counts an utterance wrong if it contains any error at all. They are one formula with three units, and they do not rank systems the same way. WER under-weights short utterances because long ones dominate the denominator; SER saturates near 100% on long-form audio; CER tracks WER when errors are whole words and diverges when they are partial. Pick the metric that matches what a failure costs your product.

How to do it

The three metrics are one formula with three units

All three are Levenshtein distance, with substitutions, insertions, and deletions each costing one, divided by the length of the reference in the relevant unit. WER divides by reference words, CER by reference characters, SER by utterances — and each utterance contributes 0 or 1 regardless of how many errors it contains.

That last difference is the largest. SER discards the error count entirely, so a system with 40% WER spread evenly over every utterance and a system with 40% WER concentrated in a third of the utterances can have SER values of 100% and 33%. They are different products.

Where each one misleads

Each metric hides a specific kind of failure, and the shape of the hiding follows from its denominator.

  • WER: a 50-word utterance with 5 errors contributes 5/50 = 0.1, the same as a 5-word utterance with 0.5 errors. Short commands carry almost no weight, so WER under-reports failure on the utterances a voice-command product cares about most.
  • CER: the right unit for languages without word boundaries (Chinese, Japanese, Thai) and for spelling-sensitive tasks. But it flatters a system that gets short words wrong: a wrong letter in a 9-character word is 1/9 of a word error, while a wrong letter in a 2-character word is 1/2 of a word error.
  • SER: the honest number for command-and-control, where a wrong sentence means the user repeats it. It stops discriminating on dictation, where nearly every long utterance has at least one error, so it sits at 99% for systems that differ by 10 points of WER.

The ranking inversion, worked through

The concrete case: system A makes five errors, all in the two longest utterances. System B makes three errors, all in the three shortest, and transcribes everything else perfectly. A has the worse WER, because five errors over thirty-eight words is a larger fraction than three. A has the better SER, because two of its eight utterances are wrong while three of B's are.

Neither system is buggy and neither metric is broken. They weight different things, and the choice is a statement about the product. If the surface is voice commands, B is worse for the user despite the better WER. If the surface is dictation, the same utterance costs the user 30 seconds either way, and A is worse. The buy decision follows: if a vendor quotes only WER on a command-style product, ask for SER before you sign.

Report counts, and an interval, alongside the percentage

A WER of 8.0% means nothing without the number of reference words. As a rough lower bound, treat errors as independent and use the binomial interval: 1.96 x sqrt(p(1-p)/N). At 2,000 reference words that is plus or minus 1.2 points; at 200,000 words, 0.12 points. The real interval is wider, because errors cluster inside utterances — a bad recording contributes several at once.

The defensible interval comes from a bootstrap: resample the utterances with replacement a thousand times, recompute WER each time, and report the 2.5th and 97.5th percentiles. It is fifteen lines of code and it accounts for the clustering automatically. Any comparison of two systems whose intervals overlap is not a comparison.

The normalizer is part of the metric

Before any of the three metrics is computed, both sides go through a text normalizer: case folding, punctuation, number and date formatting, and the policy on filler words. Each decision moves WER on the same system output, and the spread between the strictest and loosest reasonable policy is 2 to 5 points on conversational data.

Freeze the normalizer, version it, and publish it with the number. A WER comparison between two systems evaluated under different normalizers is measuring the normalizers. The same applies to the reference: the more heavily normalized reference starts several points ahead.

Code

WER, CER, and SER on one test set for three systems, including a pair that ranks in opposite orders under WER and SER.

import jiwer

REFS = [
    'yes',
    'no',
    'call mom',
    'turn on the kitchen lights',
    'set a timer for ten minutes',
    'play the next song',
    'what is the weather in chicago tomorrow morning',
    'add milk eggs and bread to my shopping list',
]

# A: five errors, all in the two longest utterances.
SYS_A = [
    'yes', 'no', 'call mom', 'turn on the kitchen lights',
    'set a timer for ten minutes', 'play the next song',
    'what is the weather in chicago',
    'add milk and bread to my shopping list',
]

# B: three errors, all in short utterances. The long ones are perfect.
SYS_B = [
    'yes please', 'no thanks', 'call mom now',
    'turn on the kitchen lights', 'set a timer for ten minutes',
    'play the next song',
    'what is the weather in chicago tomorrow morning',
    'add milk eggs and bread to my shopping list',
]

# C: one character wrong, inside one long utterance.
SYS_C = [
    'yes', 'no', 'call mom', 'turn on the kitchen lights',
    'set a timer for ten minutes', 'play the next song',
    'what is the weather in chicargo tomorrow morning',
    'add milk eggs and bread to my shopping list',
]

for name, hyps in (('A', SYS_A), ('B', SYS_B), ('C', SYS_C)):
    w = jiwer.wer(REFS, hyps)
    c = jiwer.cer(REFS, hyps)
    s = sum(1 for r, h in zip(REFS, hyps) if r != h) / float(len(REFS))
    print('system %s   WER %.3f   CER %.3f   SER %.3f' % (name, w, c, s))

print('reference words %d, reference characters %d, utterances %d'
      % (sum(len(r.split()) for r in REFS), sum(len(r) for r in REFS), len(REFS)))
print('binomial half-width at this size: plus or minus %.1f WER points'
      % (196.0 * (0.08 * 0.92 / sum(len(r.split()) for r in REFS)) ** 0.5))
  • pip install jiwer. No audio and no model needed; the transcripts are inline so the arithmetic is fully visible.
  • Expected output: system A at WER 0.132 with SER 0.250, system B at WER 0.079 with SER 0.375. B wins on WER and A wins on SER — that inversion is the point of the page, and it comes from where the errors sit, not from how many there are.
  • System C is a single character error in one long utterance. Its WER is 0.026 and its CER is around 0.006, a factor of roughly four, because a word and a character are different units.
  • The last line computes a binomial interval on 38 reference words purely to show the arithmetic. At that size the interval is enormous, which is why a real evaluation needs thousands of words or a bootstrap over utterances.

Where this goes wrong

Choosing the metric after seeing the numbers

It is easy to run all three and report the one that flatters the current system, and it is very hard to notice when it happens on your own project. Decide before the evaluation which metric is the decision metric — based on what a failure costs the user — and report the others as context. If the decision metric and the context metrics disagree, that disagreement is a finding worth explaining, not a reason to switch.

Comparing WER across different normalizers

Case folding, punctuation removal, and number formatting each move WER by a point or more on the same output, and the choices compound. A system evaluated with punctuation stripped can beat a system evaluated with punctuation scored, on identical transcripts. Publish the normalizer with the number, and re-score both systems with your own normalizer before comparing them.

Reading a WER difference smaller than the interval

Two systems at 7.9% and 8.2% WER on 3,000 words are indistinguishable — the interval is wider than the gap. This is the most common way a metric misleads: not by being wrong, but by being read with more precision than it supports. Compute the bootstrap interval, and if you cannot, at least compute the word count and do the rough binomial arithmetic.

Using SER on long-form audio

On paragraphs and meeting recordings, any real system has at least one error per utterance, so SER sits at 95 to 100% for every system and stops ranking anything. SER is informative only when utterances are short enough that a correct utterance is a realistic outcome — commands, wake words, short queries. Outside that range it is a constant.

Scoring filler words and disfluencies as errors by default

Whether "uh" counts as a word, and whether a dropped filler is a deletion, changes both the numerator and the denominator. Two teams can evaluate the identical system on the identical audio and differ by several WER points purely on this policy, and conversational speech has enough fillers for it to matter. It is a product decision — a transcription product probably wants them, a command product probably does not — but it has to be written down.

When to buy the data instead of building the pipeline

Buy when you need a test set large enough for the metric to mean something: tens of thousands of reference words with a consistent normalizer and a stated filler-word policy, which is more than most teams can annotate in-house. Buy when you need the failure modes represented, not just the volume — accented speech, noisy conditions, code-switched utterances, and the long utterances that dominate a WER denominator. Buy when the number is going into a contract, a benchmark claim, or a vendor comparison, because those need a reference with a published protocol and an inter-annotator agreement figure. If you are comparing two of your own model checkpoints during development, a few hours of your own audio is enough and the absolute numbers do not matter — only the ordering, and that survives a small test set.

How buying training data works →

More technical reference

All technical reference →

Need data for ASR Evaluation Metrics?

Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.

  • Pilot batch before the full run, so problems surface early.
  • Consent documentation delivered with the data.
  • No medical or clinical data. No recorded telephone calls.

We reply within two business days. Your details are used only to answer this request. See our privacy policy.

Contact

Talk to a human

Send a specification and we will come back with a real number and timeline.

Submit a sourcing request

Or email hello@linguacorpus.com