When to stop collecting data

Three different stops

When to stop is not one decision. Stopping the current batch, stopping the program, and stopping collection for this model generation are three decisions with different evidence, and conflating them produces either premature endings or open-ended orders.

The batch stop is a scheduling question. The program stop is a budget question. The generation stop is a technical question about whether more of the same data can still move the metric. Only the third one gets discussed, and it is the one where teams hold the weakest evidence, because that evidence has to be produced deliberately.

The measurement floor comes first

Before any curve can be read, the noise band has to be known. Train the same configuration twice with different random seeds and no change to the data, and measure the difference in the evaluation metric. That difference is the smallest effect this pipeline can detect.

The consequence is blunt. If doubling the data moves the metric by less than the seed-to-seed difference, the experiment has not shown anything. Teams chase improvements inside that band for months.

The band is usually wider than people expect, and it narrows when the evaluation set is larger and training is more stable. Narrowing the band is often cheaper than collecting the data it would take to beat it, which is a reason to invest in evaluation before investing in more hours.

Watch where the errors move, not only how many

Error counts flatten before error causes do, and the causes carry more information. Classify the current model's errors on a held-out sample by cause, and read the mix rather than the total.

A coverage gap looks like this: the model fails on material resembling nothing in the corpus — an accent, a recording condition, a vocabulary domain. More collection of the same kind will not fix it, but collection of the missing kind will, and that is a different order.

A capability limit looks different: the model has heard this kind of material and still gets it wrong, or the errors cluster on acoustically similar words or on inconsistently labelled examples. More hours will not help. The fix is annotation quality, a decoding change, model capacity, or a training recipe, and each of those is a separate budget line.

The stop signal is the point at which the dominant error class has moved from the first category to the second.

Measure how much of the new data is new

Volume is not novelty. A batch can be large and add almost nothing, and this is measurable before the money is committed.

• Nearest-neighbour distance. For each new utterance, compute the distance to its closest neighbour in the existing corpus using a speech embedding or a text n-gram model. The fraction of new material below a chosen distance threshold is a duplication estimate.
• Speaker overlap. Compare hours per speaker in the new batch against the existing distribution. Hours added to speakers who are already heavily represented are worth far less than hours from new speakers, and this is visible in the manifest rather than in the audio.
• Condition coverage. Tabulate the new batch against the conditions the specification cares about: device, environment, speaking style, dialect. A batch that repeats the existing conditions is a bigger version of the same dataset.
• Vocabulary and topic novelty, for annotation-heavy work. If the new material uses the vocabulary the old material already covered, it teaches the model little it does not have.

The marginal unit keeps getting more expensive

The last part of a collection is the most expensive part, and the reason is structural rather than commercial. The easily reached speakers were reached first. The convenient recording conditions were used first. The remaining hours need more recruitment effort, longer travel, or a population that is harder to find, and each of those adds time before it adds cost.

So the two curves cross in a predictable place: improvement per added unit falls while effort per added unit rises. When both are moving in the wrong direction at once, the value of the next batch has turned, and continuing becomes a decision to pay more for less.

Write the stop condition before the batch

The reason to write it in advance is that the decision is otherwise made mid-project by people who have already spent the budget and are invested in the answer.

• Name the metric and the evaluation set. Not a general sense of quality, but a specific number computed on a specific set that has been frozen.
• Name the increment. What the next tranche is, in units, so that the comparison is between two defined states rather than between a state and a feeling.
• Name the threshold, and set it above the seed-to-seed noise band measured earlier.
• Name the ceiling in units rather than in currency, so the maximum is expressed in the same terms as the deliverable.
• Name who decides, and name the date the decision happens. A stopping rule evaluated whenever somebody remembers is not a rule.
• Keep the unspent budget in reserve. The failure modes that only appear after deployment usually need one slice nobody could have specified at the start, and keeping recruitment channels warm costs far less than rebuilding them.