Building an ASR evaluation harness you can re-run next quarter

The difference between a score and a harness is whether you can produce the same number again in three months and explain it. A notebook that loads audio, runs a model and prints a float is not a harness. It is a result that cannot be audited.

A harness is four artifacts: one manifest, one frozen split, one cached set of hypotheses, and one report file per run. Build them in that order, because each one constrains the next, and the order is also the order in which they go wrong.

The manifest is the harness

One table, one row per utterance, with a stable identifier. The columns that earn their place: utterance id, audio path, reference text, speaker id, duration in seconds, language, and source. Paths relative to a single root so the manifest survives being copied to another machine.

Two decisions in the manifest prevent most silent failures later. First, keep the reference text inside the manifest rather than in a separate file indexed by position; index drift is the most common way a harness ends up scoring hypothesis 41 against reference 42. Second, store a sha256 hash of the manifest file in every report, computed over the bytes with hashlib.sha256(open(path, "rb").read()).hexdigest(). That hash is how you prove which version of the reference produced a historical score.

Split by speaker, before you look at anything else

Split on speaker id, not on utterance. Shuffling utterances puts the same voice on both sides, and the resulting test score is partly a measure of how well the model memorized that person. Group the manifest by speaker, assign whole groups with a fixed random seed, then assert that the intersection of the two speaker sets is empty.

There is a second leak that a speaker split does not catch: the same speaker reading the same script twice, once in training and once in test. Deduplicate on the pair of speaker id and normalized reference text before splitting, and keep one of each pair.

Freeze the split into the manifest as a column rather than computing it at run time. A split that is regenerated on each run will drift the moment a file is added, and then two reports are not comparable even though nothing about the model changed.

Inference: batch it, cache it, and keep scoring separate

Load audio at the rate the model expects. For Whisper-class models that is 16 kHz, and librosa.load(path, sr=16000) is the shortest correct way to get there. Resampling with the wrong filter is a quiet accuracy loss that shows up as unexplained errors on high-frequency consonants.

• Batch with the transformers pipeline: pipeline("automatic-speech-recognition", model=name, device=0), passing lists of arrays rather than one file at a time.
• Set the language and task explicitly in generate_kwargs instead of letting the model detect them. Detection adds a failure mode you did not intend to measure, and it will appear as errors in the slices you care about most.
• Freeze the decoding parameters: number of beams, temperature, and whether timestamps are returned. A score without decoding parameters is not reproducible, and temperature above zero makes it non-reproducible by design.
• Write hypotheses to one JSONL file per run, keyed by utterance id and model revision, so that re-scoring never re-runs inference.

Scoring: one join, one normalizer, one place

Scoring is a join between the hypothesis file and the manifest on the utterance id, followed by one call to the metric with one normalizer applied to both sides. Then assert two things before you print anything: the join produced exactly one row per utterance, and every manifest row has a hypothesis.

Report the file accounting as a first-class number: manifest rows, scored rows, and skipped rows with a reason for each. A harness that silently skips the twenty files that failed to decode reports a score for a different test set than the one on the label. If six percent of files are skipped but those files are thirty percent of the audio hours, the run is not ninety-four percent complete.

The report: machine-readable first, table second

Write one JSON file per run, named by run identifier, containing a timestamp, the code revision, the manifest hash, the model revision, the decoding parameters, the normalizer version, and the metric block. Generate the human table from the JSON rather than by hand, so the two cannot disagree.

Include the slice breakdowns from the first version of the harness, even before anyone asks for them. Retrofitting a breakdown after a decision has been made is how a team ends up defending a number it cannot decompose. Speaker, duration band and language are three group-bys that cost nothing once the columns exist.

What the harness buys you

With the four artifacts in place, a vendor delivery, your own fine-tune and a hosted model upgrade all enter through the same door. The numbers are comparable because everything except the system under test was held fixed, and the one thing that did change is recorded in the report.

The alternative is the situation most teams start in: three scores from three months, each computed with a different normalizer on a different subset, and a decision to make. Buying a delivery you cannot re-score means accepting a number you cannot check.