ASR confidence scores: what they mean and when to send a human

Where the number comes from

A confidence score is a by-product of decoding. At each step the decoder holds a distribution over possible tokens that combines the acoustic model with the language model. A word score aggregates its token scores, often as a geometric mean or a minimum, and an utterance score aggregates the word scores again. Speech models in the Whisper family expose related fields directly: the average log-probability of the decoded tokens, a no-speech probability, and a compression ratio that catches looping output.

None of those is a calibrated probability that the word is correct. They are scores from a search process, shaped by the language model, and they are overconfident by construction on the kind of audio the model was trained on. Treating a raw score as a probability is the first mistake, and it is the one every downstream threshold inherits.

Three ways the score lies

The first is the fluent hallucination. Feed silence, music, or a badly clipped recording and the decoder does not produce nothing; it produces plausible text, sometimes with a high average log-probability, because the language model is confident about fluent sentences even when the acoustics are empty. The fields that catch this concern the audio rather than the text: the no-speech probability and the compression ratio.

The second is domain shift. A score of nine tenths on in-domain audio and a score of nine tenths on accented, noisy, far-field audio do not mean the same thing, and one threshold flags a small share of the clean set and a large share of the noisy one. Any threshold calibrated on clean audio and then applied to noisy audio is really a different threshold.

The third is aggregation. A confident utterance score can hide one low-confidence word, and that word is often the number, the name, or the term the downstream system needed. If the score is going to drive a decision, keep it at the word level and carry the timestamps, so a reviewer can be sent to the right place in the audio.

Calibrate on your own audio before setting a threshold

A threshold is a claim about error rates, so it needs evidence. Take a sample of your deployment audio — a few hundred utterances is enough to start — have the references checked by a human, score the sample, and plot the actual error rate against the score. That curve is the only defensible basis for a threshold, and it is specific to the domain, the channel and the model revision.

A light calibration model turns the raw score into an expected error rate: a monotone fit such as isotonic regression, or a logistic regression over a handful of features, including the score, the no-speech probability, the duration, and a rough signal-to-noise estimate. After that the threshold reads as a statement about errors — flag everything whose expected error rate is above some level — instead of an arbitrary number, and it can be re-derived when the model is updated.

Build the review queue from signals, not one score

The queue is where confidence earns its keep. A single threshold on one score wastes reviewer time and misses the errors that matter. Combine the signals that are known to correlate with costly mistakes.

• Word confidence, kept at the word level, with the timestamp attached.
• Token type: numbers, dates, currency and measurements are error-prone and expensive when wrong.
• Names and entities, which fail more often than ordinary words and cannot be recovered from context.
• Audio conditions: far-field, noisy, or a long stretch of silence before the utterance.
• The no-speech probability and the compression ratio, which catch the hallucination class that text-based scores miss.

Measure the queue, not the model

The number that justifies the whole arrangement is the share of all errors caught by reviewing the worst slice of the batch. Compute it on a scored sample with checked references: rank by the queue order, then measure how many errors sit inside the top share you can afford to review. That curve tells you whether the queue is working, and it moves when the model, the domain or the audio conditions change.

Report the false-alarm side as well: the share of flagged words that turned out to be correct. A queue that is mostly false alarms trains reviewers to skim, and a skimmed queue catches fewer errors than its recall number promises. And keep raw confidence out of any downstream threshold — redaction, routing, escalation — until it has been calibrated for that specific use, because the meaning of a raw score moves with every model update.