Labeling robot trajectories: instructions, success labels, and how fine to go
Writing instructions a policy can use, deciding what counts as a failure, and what it costs to label at episode, phase, or frame granularity.
Robot annotation differs from most other annotation work in one way that drives everything else: the label and the action have to line up in time. A transcript can be corrected in review without touching anything downstream. A phase boundary placed half a second late teaches the policy to act at the wrong moment, and the error is baked into training.
Write the instruction as a goal, not a description
The instruction should say what to achieve, not what the video shows. "Put the can in the bin" is a goal. "Move right, close the gripper, then lift" describes the trajectory and gives the policy nothing it cannot already observe. Keep one template per task and vary only the parts that should vary — object, target, color, size — because if two annotators write the same task two different ways, the policy learns to treat phrasing as a task difference.
Test the instruction without the video. Show the text alone to someone who has not seen the episode and ask what should happen; if they cannot describe it, the instruction is underspecified, and no amount of annotation volume will fix it. For bimanual or multi-stage tasks, decide whether you want one instruction per episode or one per stage. Both are defensible, and mixing the two conventions inside one dataset is not.
Success is not always binary
An episode label needs at least three values: success, failure, and invalid. Invalid covers episodes where the operator intervened, where the object was dropped and re-grasped mid-episode, or where the scene was disturbed from outside the task. Those episodes are not negative examples, they are not examples at all, and labeling them as failures teaches the policy to avoid competent behavior.
For failures, record why. A grasp that slipped, a target that was misidentified, and an operator mistake are three different signals, and only the first two belong in a recovery set. Define success by the outcome state rather than the motion: "the can is in the bin and the gripper is open" is checkable by a second person, while "the robot did the task correctly" will be read differently by two annotators on exactly the ambiguous episodes that matter.
Pick a granularity and price it honestly
Granularity is the main cost driver in trajectory annotation, and each tier roughly multiplies the effort of the one below it.
- Episode label. One instruction and one success value per episode. Fastest by a wide margin, and enough for behavior cloning and for filtering.
- Phase labels. Timestamps for the stages of the task, such as approach, grasp, transport, place, release. Costs several times an episode label, because the annotator scrubs through the video rather than watching it once.
- Per-frame progress or reward. A continuous value for how far through the task each frame sits. The expensive tier, needing multiple passes and a written scale, and worth it only for value-based or progress-based training.
- Grasp events. The frame where contact begins and the frame where the object leaves its surface. Cheap to add on top of phase labels and disproportionately useful for diagnosing failures.
Make agreement measurable
Double-label a slice of every batch, at a rate around one episode in ten, and compare. For success labels, report the raw disagreement rate. For phase boundaries, define a tolerance before measuring, since exact frame agreement is unrealistic; a window of a few tenths of a second is usually the right scale for manipulation.
Disagreements cluster rather than scattering. In manipulation they concentrate on the moment of first contact and on whether a slip counts as a re-grasp. A guideline that defines those two cases explicitly fixes most of the disagreement, which means the fix is a paragraph rather than more training.
Tooling decides the cost more than the guideline does
A scrubber with keyboard shortcuts, a joint velocity plot alongside the video, and the ability to jump to the previous labeled boundary cuts labeling time more than any revision to the guideline. Pre-segment episodes automatically using motion features — the points where joint velocity drops to near zero are good candidates for stage boundaries — and have the annotator confirm or move them instead of placing them from scratch. The annotator's judgment is still the label; the machine only proposes.