Computing WER in Python: what jiwer rewrites before it counts

Two teams score the same model over the same audio and publish different word error rates. Neither script is broken. They handed different text to the scoring function, because every WER library rewrites the input before it counts, and those rewrites are part of the metric whether you chose them or not.

jiwer is the library most Python projects reach for, and its defaults are deliberately minimal. That is the right design for a library and the wrong default for a report. The fix is a short transform chain, and the reason to bother is that a normalization choice can move a WER further than a model change does.

Score the pair twice before you trust either number

Start with a sentence that differs from its reference only in formatting: capitalization, a trailing period, and a date written as digits on one side and as words on the other. Score it once with jiwer.wer(reference, hypothesis) and once with an explicit transform passed to reference_transform and hypothesis_transform. If the two numbers differ, you have just measured your formatting conventions.

The default chain is three steps: RemoveMultipleSpaces, Strip, and ReduceToListOfListOfWords, which splits on whitespace. Case is not folded and punctuation is not removed, so The and the are different tokens and a trailing period is a substitution. Nothing here is a bug. It is a library declining to guess what you meant.

A worked pair shows the size of the effect. The reference is "The meeting is on September 3, 2026." and the hypothesis is "the meeting is on september third twenty twenty six". As-is, that is six edits over seven reference words, a WER of 0.857. Fold case and drop punctuation and the same pair scores 0.571. The residual is the date, which no library will expand for you.

Case and punctuation are the first two knobs

The usual fix is a chain built with jiwer.Compose and applied to both sides. Order matters, and the last step is not optional: a custom transform has to end in ReduceToListOfListOfWords or jiwer raises a ValueError, because the aligner needs token lists rather than strings.

• ToLowerCase — makes The and the the same token. Standard for English; check languages where case carries meaning before treating it as free.
• RemovePunctuation — stops a comma the annotator typed from scoring as a word error. It also deletes apostrophes, so contractions still need their own rule.
• RemoveMultipleSpaces and Strip — housekeeping. Without them, a double space leaves an empty token, which the aligner counts as a deletion.
• SubstituteRegexes with a hyphen mapped to a space — the step most people leave out, and the one covered below.
• ReduceToListOfListOfWords — the terminal step. Everything before it operates on plain strings.

Numbers are the part nobody handles for you

jiwer has no number handling, and it should not: 2026 is read as twenty twenty-six, two thousand and twenty-six, or two zero two six depending on context. A model that emits digits where the reference spells them out is scored as wrong, and on a real test set that pattern repeats thousands of times, so it can dominate the headline.

The fix is to expand digits on both sides with the same function before scoring. num2words is the usual choice, and two arguments matter. num2words(2026, to="year") gives twenty twenty-six, which is how a person says a year, while the default cardinal form gives two thousand and twenty-six, which nobody says. In the worked pair, cardinal expansion leaves the score at 0.364, because the reference now reads three where the audio says third. Expanding dates with to="ordinal" and years with to="year" takes the same pair to 0.0.

That is the point of the whole exercise: identical audio, identical model output, and a swing from 0.857 to 0.0 driven entirely by text handling. Number conventions belong in the annotation guideline, and the scoring script has to reproduce them exactly. If the guideline spells numbers out, expand both sides. If it keeps digits as digits, leave them and accept that a model which spells numbers out will be penalized.

The order of the steps changes the score

Expanding a number can emit a hyphen, and hyphen handling is not cosmetic. RemovePunctuation turns twenty twenty-six into twenty twentysix, merging two tokens into one, so the aligner records a substitution plus a deletion where the model made no error. Mapping the hyphen to a space first keeps three tokens intact. Do that with SubstituteRegexes before the punctuation step, or expand numbers after it.

The same class of bug appears with any transform that changes token counts. A cheap habit is to print the token lists for three or four utterances after the transform and read them. The score will not warn you that the transform is wrong. It will simply be wrong, quietly, and usually in the direction that flatters the model.

Record the transform next to the score

A WER without its transform is not reproducible. Keep the chain in one named object in one module, pass the same instance to both sides, and write a version string for it into every report.

Two things are worth pinning while you are there. jiwer renamed its transform arguments between major versions, so an unpinned jiwer means last quarter scores are not comparable to this quarter without a re-run. And when you compare two systems, compare their transform chains first: a five-point gap is routinely a punctuation convention rather than an acoustic difference.