Annotating robot episodes at scale: throughput, sampling, and what not to label

At scale, the question is what not to label

A pilot can afford to label everything. A collection of tens of thousands of episodes cannot, and the projects that try usually spend most of the budget on the easy majority: short, clean, obvious successes that a policy learned from long ago. The structure that works is a funnel rather than a uniform pass.

Every episode gets an automated outcome. A sample of episodes gets human labels. A smaller slice gets frame-level work. The expensive treatment goes to the tail: long-horizon tasks, near-failures, rare object classes, and the conditions where the policy performs worst. Which slice is the tail is a question the evaluation results can answer, and pointing the annotation budget at the worst-performing conditions is the highest-yield allocation available.

Automate the outcome label first

Success is often measurable rather than judgeable. The final object pose, a known terminal position, the gripper state, a joint torque signature, or a scripted scene check can each decide whether an episode ended correctly, and a vision model can propose a label for human confirmation. Confirming is faster than deciding, and it is more consistent across annotators because the judgment is narrowed to a binary agreement.

Keep the automatic label and the human override in separate fields. That pair is what lets you measure how often the automation is wrong per task, and it is what makes it safe to widen automation later: a task where the checker agrees with human review ninety-nine times in a hundred can be trusted at scale, and one where it agrees eighty times cannot.

Throughput is bounded by watching and scrubbing

Episode-level labeling is bounded by watching time. An annotator who has to see an episode end to end spends at least the episode duration plus overhead, so a fifteen-second episode yields something on the order of a few hundred episodes per annotator-day before quality suffers. Frame-level work is bounded by scrubbing instead, which is slower by a multiple rather than by a percentage, and that difference is why the granularity decision is the main budget decision in the whole pipeline.

The largest throughput lever is pre-segmentation. Motion features give candidate boundaries, since joint velocity collapsing to near zero marks the end of a stage, and a depth or contact signal marks a grasp. The annotator then confirms or moves a proposal instead of placing a boundary from scratch. A video scrubber with keyboard shortcuts and a velocity plot beside the frames is the next lever, and it is worth more than another revision of the guideline.

Sampling and agreement at scale

Double-label a fixed fraction of every batch, sized so that agreement can be estimated per task rather than only across the whole set, and do it continuously rather than in a single audit at the end. Agreement measured per task and per annotator is what shows where the guideline is unfinished, and the disagreements cluster rather than scatter: in manipulation they concentrate on first contact and on whether a slip counts as a re-grasp. Each cluster is a paragraph in the guideline, not a retraining problem.

Add a gold set to every annotator queue: a small set of episodes with known labels, injected at a rate low enough not to be noticed and containing the ambiguous cases rather than the obvious ones. A gold set made of easy examples measures nothing, because everyone gets them right. The one built from the disputed cases detects drift over weeks, which is the failure mode that a single audit misses entirely.

Keeping a large annotator pool aligned

Guideline changes are versioned and pushed to the whole pool at once, with the gold set re-run immediately after each change, because a guideline that reaches half the pool is two guidelines. At scale, batch ownership beats central review: one reviewer owns one batch, and their decisions are logged as precedent so that the same ambiguity is resolved the same way next month. A single central reviewer becomes the bottleneck within weeks.

Churn is the other constant. A pool always contains recent joiners, and their first episodes should be review-only until their agreement against the gold set stabilizes. Track agreement per annotator over time rather than as a single number, since a drop in one annotator agreement usually means a guideline ambiguity surfaced rather than a person getting worse.

Labels ship as a versioned artifact

Labels belong in a sidecar keyed by episode identifier, with a schema version, never written into the raw episode files. A label change produces a new version rather than an edit in place, so any training run can be reproduced against the labels it actually used. Uncertain and rejected decisions are part of the record as well: no label is a decision, and losing it makes the batch look complete when it is not.

When the guideline changes, do not re-label everything. Re-label the gold set and the affected slice, record which version each episode carries, and let the rest keep their version. A dataset where every episode records the label schema it was annotated under is one that can still be trained on selectively a year later, and that is the property that makes a large annotation effort reusable instead of disposable.