How big an ASR test set needs to be, and how to build it
Word count sets the noise floor, but clustered errors mean the effective sample size is closer to the speaker count. Here is how to compute both numbers.
Two questions decide whether a test set can support a decision. How big is it, which has an arithmetic answer, and what is it made of, which has a procedure. Teams usually answer the second one by instinct and never ask the first.
The arithmetic matters because a test set that cannot distinguish two systems will still be used to choose between them. An interval that spans the difference is a coin flip with a decimal point.
The arithmetic: what a one-point difference costs
Treat the pooled error rate as a proportion over reference words. The standard error is roughly the square root of p times one minus p, divided by the number of reference words. At an error rate of 0.10, putting a ninety-five percent interval of about one point around a single score needs roughly 3,600 reference words.
Comparing two systems is worse, because both estimates carry noise. To resolve a one-point difference between two systems at that error rate, budget about 7,200 reference words for each one. For a two-point difference, divide by four, since the sample size scales with the square of the precision: about 1,800 words each.
This is a floor, not a plan. The calculation assumes errors are independent across words, and they are not. The next section is about how far off that assumption leaves you.
Clustered errors: why the word count is optimistic
Errors arrive in runs. One difficult speaker produces errors on most utterances, one noisy recording produces errors across its whole length. When errors cluster, the effective sample size is smaller than the word count, and the interval is wider than the arithmetic above suggests.
The standard correction is a design effect: one plus the cluster size minus one, multiplied by the intra-cluster correlation. Take 4,000 reference words spread over 20 speakers, so 200 words each, and an intra-cluster correlation of 0.1. The design effect is about 21, which makes the effective sample size roughly 190 words and the ninety-five percent interval about four points wide instead of one.
The practical consequence is that the speaker count, not the word count, is the lever. The same 4,000 words spread over 40 speakers instead of 20 halves the cluster size and narrows that interval from about four points to about three. Adding words to the existing speakers barely moves it.
You can estimate the correlation from your own data rather than assuming it. Compute the pooled error rate per speaker, take the spread across speakers, and compare it to the spread you would expect from independent errors at the same per-speaker word counts. If the observed spread is much larger, errors are clustered and the design effect applies. Reporting the per-speaker spread is worth doing anyway, because it is the number that tells you whether the score will hold on the next batch of audio.
Stratify on the things that move the error rate
Composition is a procedure, and it starts from a list of the variables that change accuracy: speaker, duration band, channel or device, background noise, speaking style, and the first language of the speaker. Pick the ones that vary in production, and make sure each is represented by more than one speaker, since a stratum represented by one person is a biography rather than a sample.
The usual advice is to mirror production proportions. That is right for the headline and wrong for the tail. Deliberately oversample the strata where failure is unacceptable, then report both the production-weighted number and the raw per-stratum rows. Oversampling is cheap; discovering the tail after launch is not.
A stratum needs enough words to produce a number you can act on, which by the arithmetic above is a few thousand words for a tight interval. Most sets cannot afford that in every stratum, so the honest alternative is to report wide intervals per stratum rather than a single number with implied precision.
Keep two sets, and say which is which
A development set exists to be tuned against. It can be large, it can be refreshed, and its score can be quoted internally without ceremony. A locked set exists to be run on frozen candidates, and its rules are different: no speaker appears in training data, no decision is made from a partial run, and every evaluation is logged.
Replace a development set when its distribution drifts away from production by more than the interval around its score, or when consecutive model versions produce scores that are indistinguishable on it. A saturated set cannot rank candidates, which makes it worse than useless for the decision it is being used for.
Replace a locked set when it has informed enough decisions that its score has become optimistic, which is a count of evaluations rather than a date. Building the replacement before you need it is the only way to avoid a quarter of arguing about which of two contaminated numbers is real.
What to ask for when you are buying the data
If you are buying evaluation data, or accepting a model evaluated on someone else data, ask for four things: the manifest with speaker identifiers, the split definition, the reference version, and the normalization used for scoring. All four are cheap to produce for anyone who built the set deliberately, and their absence is informative.
Then ask the direct question: is any speaker in this test set present in the training data of the system being evaluated? A vendor who can answer that with a procedure rather than an assurance is selling you something you can check.
The size follows from the decision you need to make. Pick the difference that would change your choice, compute the words it requires, then check the speaker count against the design effect. The second number is the one that is usually wrong.