How to define acceptance criteria for a dataset

An acceptance procedure has four parts

Acceptance criteria are usually written as a quality target, which is why they are hard to apply. A target describes what you hope for. A procedure ends in a decision, and the decision needs four parts: what is measured, what gets inspected, how long you have to inspect it, and what happens to what fails.

A procedure missing any one of the four produces a predictable outcome. No metric means the argument is about impressions. No sampling rule means the inspection is not representative. No window means acceptance happens by silence. No disposition rule means rejected material sits in limbo while the schedule moves on without it.

Define the metric so two people compute the same number

A criterion is usable when someone else, given the same files and the same procedure, arrives at the same value. That requires a formula, a normalization step, and a reference. State the unit the metric applies to as well, because a per-batch average hides a batch that is half perfect and half broken, and that batch is the one that damages the model.

• Transcription accuracy becomes word error rate, with the normalizer written out: casing, punctuation, numerals, and how a filler word is treated on each side.
• Audio quality becomes a measured noise floor plus a rejection rule for clipping and dropouts, rather than a word such as clean.
• Label consistency becomes an agreement figure on a double-annotated subset, with the agreement method named so the number can be recomputed by anyone.
• Format compliance becomes a validation script that runs on the delivery and returns pass or fail, with the schema it checks published in advance.

Decide what gets inspected

Sampling decides whether the acceptance decision is honest. Three layers, used together, cover the cases that matter. Two conditions apply to all three: the draw has to be made by you or by a process you can verify, since a sample selected by the party being sampled is not a sample; and the sample size is a judgment worth stating in terms of what it is meant to catch, so a sample sized to detect a defect rate at twice the allowed level is a defensible statement while a spot check is not.

• A random draw, so the estimate of the defect rate is not selected for you.
• A risk-stratified draw covering every stratum and condition in the specification, because rare strata are where failures concentrate and a purely random draw may miss them entirely.
• Everything the producer flagged as borderline. A flag is free information about where the weak material is, and ignoring flags in the name of randomness discards the one honest signal the producer provides.

Set the acceptance window on purpose

Most contracts contain a default that silence means acceptance, and the window attached to it is usually short. That default is fine as a default and dangerous as an accident, because the party with more review capacity wins whichever way it points.

Name four things: the length of the window; the event the clock starts on, which should be transfer complete and manifest reconciled rather than the word delivery; who is permitted to inspect; and what happens if you miss your own window. A procedure that lets the buyer hold a batch indefinitely is one producers price for, and the price shows up in the unit rate.

A short window is workable when the sampling plan is written in advance. It is not workable when the plan is to look at the material whenever there is time.

The disposition of rejected units

Every rejection ends in one of four dispositions, and the contract should name all four so the choice becomes a decision rather than a negotiation. One boundary rule prevents a category of dispute: when a unit fails for a reason the specification never covered, the specification gets a sentence, which makes it a guideline change rather than a renegotiation.

• Repair: the unit is corrected and re-inspected against the same criterion. Re-inspection is not optional, because a fix that is never verified is an unverified fix.
• Replacement: the batch is replaced with new material, which is the right disposition when the failure is systematic rather than item-level.
• Acceptance with a recorded deviation. Sometimes this is correct: material usable for a narrower purpose should be recorded as such rather than argued about, since the record is what keeps a deviation from becoming the standard.
• Rejection with a credit and no repair, appropriate when repair costs more than the material is worth. It should be stated as an available option rather than discovered case by case.

The defaults that quietly accept bad data

These arrangements look like acceptance procedures and are not. Against them, keep the acceptance record: what was inspected, what failed, what was accepted with a deviation, and the reason for each. That record defends the decision later, and it becomes the input that makes the next set of criteria sharper than this one.

• A per-batch average with no per-unit view of the same material.
• An inspection that reviews the producer's best examples, or the producer's own quality report, with no independent draw.
• A spot check with no sampling rule, which in practice inspects whatever is easiest to open.
• No re-inspection of repaired material, so a fix becomes a claim rather than a verified state.
• A criterion that exists only as an adjective, which cannot be failed and therefore cannot be enforced.