Five ways a WER ends up better than the system really is

An inflated error rate is almost never the result of dishonesty. It is the result of five ordinary behaviors in an evaluation pipeline, each of which is invisible from the score itself and each of which biases in the same direction.

Here they are in the order they tend to appear, with the check for each. All five checks read files you already have, and together they take less time than a single training run.

• Files that were never scored.
• Overlap between the test set and the training data.
• How many times the test set has already been run.
• The reference changing after the hypotheses were generated.
• A domain mismatch that runs in the flattering direction.

One: the files that did not get scored

Every harness skips something. A decode failure, a file with no transcript, a duration filter applied during preprocessing, an audio loader that returns an empty array for a corrupted header. The skipped files are rarely a random sample, because the things that break are the noisy ones, the long ones and the unusual ones.

The check is a count with a reason attached: manifest rows, scored rows, and a reason for every difference. Then compute the skipped share by audio duration as well as by count. A run that drops six percent of the files but thirty percent of the hours is not a ninety-four percent run, and the number it reports describes a test set that no longer exists.

Two: overlap between the test set and the training data

Text overlap is easy to measure and usually surprising. Normalize both sides, take every sequence of thirteen consecutive tokens from the test references, put them in a set, and count how many test utterances contain at least one sequence that also appears in the training transcripts. Report the share, then report the score with those utterances excluded as well as included.

Thirteen tokens is long enough to be meaningful and short enough to catch a repeated sentence with a word changed. For sets of short commands, use five. The affected utterances score near zero, so even a small share can move a pooled number by more than the difference between two model versions.

Audio overlap needs its own check, because identical text can come from a different recording and identical audio can arrive under two file names. Hash the decoded waveform rather than the file, since the same recording re-encoded at a different bitrate has a different file hash and the same samples. Comparing duration plus normalized reference text across splits catches most of the rest.

Three: how many times the test set has been run

Every evaluation that informs a decision moves a little information from the test set into the model. That is not a flaw in anyone method; it is what selection does. Run the set forty times while tuning hyperparameters and the best score you have seen is partly the maximum of forty noisy draws.

The check is a log, not a formula: one line per evaluation, with the date, the system, and the decision it informed. When that log shows more than a handful of entries, the set has become a development set wearing a test set name, and its score is an optimistic estimate of a number you would get on fresh audio.

The fix is structural rather than statistical. Keep a development set for tuning, refreshed as often as you like, and a locked set that is run only on frozen candidates and only a few times a year. When the locked set has informed enough decisions to be suspect, retire it and build a new one. A test set is consumed by use, not by time.

Four: the reference changed after inference

A score is a comparison between a hypothesis and a reference, and if the reference was edited between the two steps, the score describes a comparison that never happened. Adding punctuation, removing fillers or fixing typos after the fact all raise the score without touching the model.

The check is a hash, recorded twice: once when the hypotheses are generated, once when they are scored. If the two hashes differ, the run is invalid rather than slightly off, and the fix is to regenerate the hypotheses against the current reference. This is also the failure that a positional join produces, where hypotheses are matched to references by row order after a file was inserted in the middle. Join on the utterance identifier and assert one row per identifier before scoring.

Five: the mismatch runs in the flattering direction

A test set recorded in a quiet room with a headset microphone will report a better number than a product that receives phone calls in a car. This is well known, and it is still the most common source of an over-optimistic number, because the size of the gap is never measured.

Measure it instead of assuming it. Take a sample of production audio, estimate signal-to-noise ratio, duration and speaking rate for both sets, and compare the distributions rather than the means. If production has a lower median signal-to-noise ratio by a wide margin, or a much larger share of utterances under two seconds, then the test score is an upper bound and the report should say so in those words.

The comparison costs an afternoon of audio analysis, and it converts an unknown bias into a stated bound, which is the difference between a number that can be defended and a number that can only be repeated.

Put the five checks in the report

Each check produces one field: files skipped with reasons, overlap share, evaluation count, reference hash, and the distribution comparison. They are five keys in the JSON report next to the metric, and they turn a score into a claim someone else can audit.

The pattern behind all five is the same. The metric is trustworthy and the pipeline around it drifts, in one direction, for ordinary reasons. Checking the pipeline is the only way to know which number you are holding.