Measuring ASR latency and throughput without fooling yourself

Accuracy decides whether a transcription product works. Latency and throughput decide whether anyone uses it, and whether the compute bill is a line item or a crisis. Both are easy to measure and easy to measure in a way that predicts nothing about production.

Five numbers cover it: the real-time factor with a stated boundary, the batch curve, the memory ceiling, the time to first word, and the accuracy cost of the decoding setting you chose. Measure them together, because they trade against each other.

Real-time factor, defined by what is inside the timer

The real-time factor is processing time divided by audio duration. Below one means faster than real time, so a factor of 0.3 transcribes an hour of audio in about eighteen minutes. The number is meaningless without its boundary, so state what the timer covers: audio decode, resampling, feature extraction, model forward pass, decoding, and any post-processing such as punctuation restoration.

The boundary is where most cross-vendor comparisons break. One number includes the audio loader and the other starts after the array is in memory, and on short clips that difference is most of the measurement. Pick a boundary, write it in the report, and keep it fixed across systems.

A single factor also hides a curve, because short clips are dominated by fixed overhead while long files amortize it. Measure in duration bands instead: up to 3 seconds, 3 to 10, 10 to 30, and over 30. Report the band that matches production traffic, and report the band edges so the reader can tell whether the numbers are comparable to theirs.

Warm up, then report a median rather than a mean

The first inference includes kernel compilation, lazy initialization and the first large memory allocation. Including it inflates the average and, worse, inflates the variance, which makes the measurement look unstable when the steady state is fine. Run three to five warm-up iterations and discard them, then time a loop with time.perf_counter and report the median and the 95th percentile.

The median matters because one slow iteration moves a mean and does not move a median. Checkpoints written to disk during the run, another process taking the GPU, or a thermal event all produce a single large outlier, and an average silently absorbs it into the headline.

If the tail drifts upward over a long run, that is thermal throttling or memory pressure rather than model behavior. The fix is a different measurement environment, and the way to catch it is to keep the per-iteration times rather than only the summary.

The batch curve is the throughput number

Throughput is audio hours processed per compute hour, and it rises with batch size until the hardware saturates. Sweep batch sizes of 1, 2, 4, 8 and 16 at a fixed clip length, and record audio seconds processed per wall second at each one. Two features matter: the knee, where the curve flattens, and the point where throughput stops improving or memory runs out.

Measure the memory ceiling explicitly. Reset the peak counter with torch.cuda.reset_peak_memory_stats() before a batch, then read torch.cuda.max_memory_allocated() after it. That tells you the largest batch the device will hold for that clip length, which is a different number from the fastest one.

Batching clips of very different lengths wastes compute on padding, so bucket by duration before batching and say in the report whether you did. And keep the two products separate: a factor measured at batch 1 describes streaming, a factor at batch 16 describes offline file transcription. They are not interchangeable, and quoting the better one for the other product is how a capacity plan ends up wrong by an order of magnitude.

Time to first word is what the user feels

For live captioning or voice interfaces, the number that matters is the delay between the end of a spoken word and its appearance on screen. It is governed by the chunk size of the audio you feed the model, and it is not visible in a real-time factor at all.

Measure it by feeding fixed chunks — 200, 500 and 1000 milliseconds are a useful sweep — and recording the wall-clock delay from the end of each chunk to the first new token, counting partial results as emissions. Report the pair rather than a single latency: chunk size and delay, plus the lookahead or context window the model was given, since a longer context lowers the delay for the same chunk size by giving the decoder more to work with.

Smaller chunks lower the delay and raise the compute, because the model reprocesses context for every chunk. That trade is a product decision, and it cannot be made from a throughput number.

Beam size is the accuracy-latency dial

Greedy decoding is the fastest setting and the least accurate. Beam search improves the output, and the decode time grows with the beam width, so a beam of five costs several times a greedy decode. The gain is also not uniform across the test set: it concentrates in the noisy slices and the smaller languages, and it is often close to zero on clean read speech.

That means the sweep is a small table with three columns — beam size, real-time factor, and per-slice error rate — rather than a single setting. An offline transcription job can afford the wider beam. A live caption path on the same model may not, and the two should be reported as two configurations.

Report the decoding parameters next to both numbers, accuracy and speed. A real-time factor without a beam size and an error rate without one are equally unquotable, because the two settings that produced them are the same decision.

The block of numbers that makes a measurement usable

Six fields make a latency claim portable: the timer boundary, the audio duration distribution with band edges, the batch size, the beam size, the hardware, and the warm-up policy. With those, someone else can reproduce the setup or explain the difference; without them, the numbers are only comparable to themselves.

Speed and accuracy are one decision, not two. Measure them in the same run, on the same audio, and publish them as a pair, because a configuration that is fast and wrong and one that is slow and right are the same product choice made twice.