Normalize before you score, and record what you normalized away
Normalization can move a WER further than a model upgrade. Run the ablation, freeze one function, and stamp its version onto every score you publish.
Every error rate is a ratio between two normalized strings. There is no unnormalized WER, because the moment you decide that a difference counts as an error you have made a normalization decision. The question is not whether to normalize. It is which steps, in which order, and how you record them.
The reason this deserves more than a paragraph in a README is the size of the effect. Case folding, filler handling and number expansion each move a score by more than a typical training run does, and they are all reversible decisions made in ten lines of code.
Run the ablation instead of arguing about it
Apply one transform at a time to the same pairs and print the score after each step. The size of each move tells you what the model is actually being penalized for, and the numbers are usually a surprise to whoever wrote the reference.
A worked case, scored with jiwer. The reference is "we should uh launch on friday i think" and the hypothesis is "we should launch on friday i think". As written, that is one deletion over eight reference words, a WER of 0.125. Remove the filler word from both sides with RemoveSpecificWords and the same pair scores 0.0.
Twelve points, from one convention. And note the direction of the error: with a verbatim reference, a model that omits disfluencies is penalized; with a clean reference, a model that emits them is penalized instead, through insertions. The same convention choice produces opposite verdicts on the same behavior, which is why the guideline and the scorer have to agree.
Three normalizations that are quietly cheating
The steps above are legitimate. These are the ones that inflate a number without anyone intending fraud.
- Choosing the transform per file after seeing which files fail. Pick one chain for the whole set, or the score is a selection effect.
- Rewriting the reference from the hypothesis. Adding a punctuation convention to the reference after the fact converts scoring into copying, and it is the single most common way a benchmark stops meaning anything.
- Dropping the files the normalizer cannot handle, then reporting the score on the remainder. If unmappable numbers or mixed scripts force files out, report how many left and why.
- Normalizing away the feature you are selling. If punctuation and casing are part of the product, folding them out of the metric makes the evaluation blind to the thing customers pay for.
The order of the steps is part of the definition
Write the chain as a numbered list and treat a reordering as a change to the metric, because it is one. A workable order for English: fold case, expand numbers and abbreviations, replace hyphens with spaces, remove punctuation, collapse whitespace, strip, then tokenize.
Two of those positions carry weight. Number expansion runs before punctuation removal, because an expansion can emit a hyphen or a comma that the punctuation step would otherwise eat or merge. And hyphens are replaced with a space rather than deleted, because deleting the hyphen in a compound turns two tokens into one, which the aligner records as a substitution plus a deletion rather than as nothing.
For languages where the reference includes diacritics, insert one more decision before the tokenizer: whether accents are scored. Folding them raises the score and hides a real defect in languages where they change meaning, so if you fold them, say so in the report.
Freeze it, version it, stamp it
Put the chain in one function in one module, apply the same object to both sides of the comparison, and give it a version string. Write the version and a hash of the module into every report next to the metric.
The reason for the hash rather than just a version number is that two hand-written functions that are supposed to do the same thing drift apart within weeks, usually by one step, and nobody notices until a re-scored delivery disagrees with a historical number. When the normalizer changes, every earlier number becomes incomparable. That is acceptable as long as the report says which version produced which score.
The honest report line
A publishable line reads: metric, unit, value, normalizer version, and the count of files scored against the count of files in the manifest. Five fields, and any of them missing turns the number into something a reader cannot check.
This is not bureaucracy. It is the difference between a score that a second team can reproduce next quarter and a score that becomes a matter of memory and goodwill.