Handling a data batch that failed acceptance

Two kinds of failure, and they are not the same problem

A batch can fail against the acceptance criteria, or it can fail against fitness for use. The first is measurable and it is a contract matter. The second is a judgement that the data, while matching what was specified, does not do the job it was bought for.

Keeping them apart matters because the responses differ. A criteria failure is the producer's to remedy. A fitness failure is usually a specification problem, which puts the cost with the buyer, and treating it as a breach ends a relationship without fixing the data.

The honest first step in either case is the same: measure the extent of the problem before deciding what to do about it.

Measure the extent, and look for clustering

The useful question is not whether the batch fails but what fraction fails and whether the failures cluster. A clustered failure has a cause and an identifiable boundary. A diffuse failure points at a process problem, and repair does not fix a process.

Cluster analysis is a cross-tabulation, and it needs dimensions to tabulate against, which is why the manifest has to carry them from the start.

• Defect rate by speaker, which isolates a mis-screened participant or one recorded under unusual conditions.
• Defect rate by session and by date, which isolates a bad room, a bad day, or an equipment change.
• Defect rate by recorder and by room, which isolates a configuration error that will recur until it is found.
• Defect rate by annotator, which isolates a guideline that one person read differently.
• Defect rate by position in the batch, early or late, which isolates fatigue in the production schedule.
• One dimension with a markedly higher rate is the cause. If no dimension stands out, the failure is diffuse and the response changes.

The three outcomes

The classification and the clustering analysis exist to answer a single question: which of three responses is the right one. Two of them are affordable and one is not, and choosing by instinct rather than by the analysis is how a recoverable batch becomes an expensive one.

• Repair, when the audio is sound and the defect is confined to identifiable files: an annotation error, a normalization rule applied inconsistently, a missing field. Repair costs less than re-recording because the expensive part, the speaker, is already captured. Two conditions apply. The repair must be done by someone other than the person who made the error, and the repaired subset must be reviewed as a subset rather than spot-checked.
• Re-record, when the defect is in the audio itself: a wrong rate, clipping that cannot be undone, a room with a hum, a speaker who was not the variety the specification required. Also when the annotation defect is diffuse, because repairing a diffuse defect means re-annotating everything and the cost converges on re-recording anyway.
• Accept with a documented deviation, when the shortfall is bounded and the data still serves the use. This needs a written, signed deviation, the affected slice marked in the manifest, and an adjustment to scope or terms. An unmarked deviation becomes an unexplained evaluation result six months later.
• A working line between repair and re-record: repair is usually right when the affected fraction is under roughly a third and the defects are clustered. Above that, or when they are spread evenly, re-record.
• What must not happen is the fourth outcome, shipping the batch unchanged and hoping nobody notices. Somebody always notices, later, and by then cannot explain it.

The clauses that make this survivable

All of this is decided long before the failure, in the contract, and the clauses are cheap to negotiate at the start and impossible to add afterwards.

• A defined acceptance procedure: the sample size, the sampling method, the metric, and the threshold. Without it, acceptance is an opinion, and the argument becomes about whose opinion counts.
• An included rework allowance: a stated fraction of the delivery that may be rejected and re-produced at no additional cost, with the terms beyond it agreed in advance. A project with no allowance has this negotiation at the worst possible moment.
• A re-collection clause: who bears the cost when the failure is the producer's, and what happens when it is the buyer's because the specification changed or was ambiguous.
• A deviation procedure: who is authorised to sign one, and the requirement that any accepted deviation is recorded in the manifest.
• A retention clause: the producer keeps the raw material and the working files for a stated period after delivery. If the raw files are deleted at handover, every future failure becomes a re-record whatever the cause.
• Milestone payments placed where failure is detectable — end of recruitment, close of the pilot review, first accepted batch — so that a failure is found at a fifth of the way through rather than at delivery.

Prevention comes from the root cause, not the blame

Every failure has a root cause, and the root cause is almost always a process rather than a person. A defect that reached acceptance means the sampling missed it, the guideline did not cover the case, or the screening let the wrong participant through. Each of those is a change to a document or a procedure, and each is testable at the next batch.

Keep the reject log with its reason codes and read it before the next batch starts. A defect that is not understood will be produced again at the same cost on the next delivery, and the second occurrence is harder to explain than the first.

There is also a relationship dimension worth designing for. A producer who reports a failure before the buyer finds it is worth keeping; one who ships and hopes is not. Writing self-report into the contract with a defined consequence, such as a rework allowance that is used rather than forfeited, turns an incentive to hide into an incentive to disclose.