How to sample for annotation QC: full inspection, stratification, and how much is enough

Why inspecting everything is the weaker plan

The intuitive QC plan is to check every item before delivery. It feels like the safe choice, and it fails for two reasons. The first is arithmetic: review time scales with the batch, so on a long project the review step becomes the bottleneck and gets quietly shortened, usually by replacing the most experienced reviewer with whoever is free that week.

The second reason is worse. One reviewer applying a consistent — and consistently wrong — reading of the guideline will pass every item, because the same misunderstanding that produced the errors will also accept them. Full inspection by one person measures agreement with that person, not correctness. The errors that damage a delivered dataset are systematic ones, and a single pair of eyes cannot see them, however many items it reads.

Choose the unit of sampling deliberately

Samples are drawn from units, and the unit decides what the numbers mean. Utterances, files, speakers, and batches are not interchangeable.

The trap is correlation inside a recording. Three hundred utterances drawn from ten recordings are not three hundred independent observations; they share a speaker, a room, and a microphone, so a defect present in one tends to be present in all of them. Forty files, sampled as files, give forty independent chances to catch a file-level defect. The same effort spent on utterances can collapse into six files if the draw is unlucky.

• Utterance level: catches transcription slips, but overstates independence and hides anything shared within a recording.
• File level: the right unit for defects that are properties of a recording — noise, clipping, a wrong sample rate.
• Speaker level: the right unit for anything about a person — accent coverage, pronunciation habits, a speaker who should not have passed screening.
• Batch level: the right unit for process defects, such as one day's material annotated under an older guideline version.

Stratify on the things that can go wrong

A uniform random draw is only as good as the batch being uniform, and batches are not. The deadline-week material, the hardest accents, the longest files, and the items the annotator flagged are all different from the average, and they are where defects concentrate.

Strata worth having in almost every project:

• Annotator — the unit that lets a reviewer tell a person problem from a guideline problem.
• Batch and date — catches a guideline change, a new team, or one bad week.
• Duration band — long items are where fatigue errors and truncation live.
• Flagged items — anything the annotator marked as borderline, plus items containing crosstalk, noise, or unintelligible speech.
• New speakers — anyone whose material has not been checked yet, at least once.

How many items: two questions, two formulas

Two different questions need two different sample sizes, and conflating them is why QC samples end up either uselessly small or absurdly large.

The first question is detection: if a defect class affects a share p of the batch, how many items must be drawn to be reasonably sure of seeing at least one? For independent draws, the sample size is the natural logarithm of one minus the confidence you want, divided by the logarithm of one minus p. At 95% confidence, a defect affecting 1% of items needs roughly 300 draws, and one affecting 5% needs roughly 60. This is the right sizing when you only need to know whether a defect class exists at all.

The second question is estimation: what is the defect rate, to what precision? For a rate near 5%, a sample of 400 gives a margin of error of about two percentage points at 95% confidence. Halving that margin takes four times the sample. This is the right sizing when the number itself will drive a batch decision.

The practical consequence: detection sizing is cheap, estimation sizing is not. A QC plan can afford to run detection on many defect classes and reserve estimation for the two or three that decide acceptance.

Write the batch decision before the sample

A sample has authority only if the decision it feeds was defined in advance. For each defect class, state the rate that is acceptable, the rate that forces rework, and what rework means: re-inspecting the affected stratum, re-briefing the annotator, or rejecting the batch outright.

Two rules keep this honest. A batch that fails is never fixed by re-sampling — the second draw differs from the first by chance, not by improvement. And every accepted defect gets a reason code and a destination, because the pattern across batches is the input to the next guideline revision.

Make the sample reproducible

Record the seed, the unit list, the strata, and the reviewer for every sample. A QC figure that cannot be reproduced is an anecdote. The record also answers the question a serious buyer eventually asks: what exactly was checked, and how was it drawn.