Human in the loop: where the person actually belongs in a data pipeline

Human in the loop gets used as if it described one thing. In practice it describes a design choice about which step in a pipeline depends on a person's judgment. Put the person in the right place and the pipeline becomes trustworthy. Put them in the wrong place and you have doubled the cost for no change in the output.

The test for the right place is simple. If a rule can check the answer — duration, sample rate, duplicate detection, file naming, an audio quality threshold — then a human review is expensive and, worse, less consistent than the rule. If the answer needs judgment that no rule can express, a person belongs there and nothing else will do.

The steps that cannot be automated away

Judgment concentrates in a small number of places: whether a boundary case belongs to one category or another, whether an answer is acceptable rather than merely plausible, whether the guideline covers the case at all, and whether a recording is fit for the purpose it was collected for.

Those are exactly the places where guidelines grow. Every ambiguous case that reaches a person and gets resolved becomes a line in the next version of the guideline. That is how a pipeline's dependence on human judgment shrinks over time without quality falling. A pipeline where the same question reaches a person in month six as in month one is not improving; it is paying a permanent tax for a decision nobody wrote down.

Pre-annotation changes what you are paying for

The most consequential design choice is whether the person writes the answer or corrects one. A model proposes; the person accepts, edits, or rejects. The unit of cost moves from producing an item to verifying one, and verification is faster — but only on the slices where the proposal is good.

That condition is the whole thing. Measure the acceptance rate per slice, because a slice where the proposal is usually wrong costs more with the model than without it. Reviewing a bad suggestion takes longer than producing the item from scratch, and it anchors the reviewer toward the suggestion's shape. A pipeline that pre-annotates everything, including the slices where the model is weakest, pays twice for the same bad output.

Sampling review, and what the sample has to be

No pipeline reviews everything, so the review sample is the quality measurement, and it has to be built like one.

Three rules keep it honest. Sample randomly within each slice rather than taking the first files or the most recent ones, because both are unrepresentative in different directions. Size the sample so it can detect the error rate you care about, since a review of twenty items cannot tell a two percent error rate from a five percent one, and that difference is usually the acceptance criterion. And keep the outcome recorded per reviewed item, so the rate is computed rather than estimated.

The bias to avoid is reviewing only what looks suspicious. A reviewer who inspects the items that draw attention finds more errors per item reviewed and reports a lower error rate for the whole delivery, because everything unremarkable went unmeasured. The number looks better and the data does not improve.

One more rule: the reviewer should not be the producer, at least for a defined share of the sample. Self-review catches typos and misses misunderstandings, because the misunderstanding is shared between the producer and their own work.

The two shapes, and the latency they cost

There are two coherent places to put the person, and they buy different things. Gating production, where the person approves before an item is final, is slower and prevents waste: nothing bad enters the batch. Gating delivery, where the person reviews after the batch exists, is cheaper per item and faster, but it finds problems at the moment when fixing them is most expensive.

Both are defensible. Choosing neither is not. The failure mode is a review step that appears on the schedule but has no authority to reject anything, which buys the cost of the gate and none of its effect.

Latency deserves explicit planning, because human throughput is the one part of a pipeline that adding hardware cannot improve. If the review step requires a person to read every item, the delivery date is set by reading speed and by how many qualified reviewers are available at the same time — not by the recording schedule.

What to pin down with a supplier

Dropped items that nobody counts make the delivered quality look better than the pipeline that produced it, which is why the last two items on this list matter more than they appear to.

• Which step the person sits at, and what that person is allowed to reject.
• The review rate, the sampling method, and whether the reviewer is different from the producer.
• The measured error rate on the reviewed sample, and how it was computed.
• The rule for resolving disagreement between a producer and a reviewer — a third adjudicator, a majority, or a rewritten guideline.
• What happens to items that fail review: reworked, or dropped, and if dropped, whether the count is reported.