Failure data: the part of a robot dataset most pipelines throw away

The default pipeline deletes the useful part

A collection pipeline filters at ingest: episodes that fail a quality check or that end in a failed attempt are dropped, and the reject log keeps a count. Filtering is right. Deleting is the mistake, and it happens for a mundane reason, which is that image-heavy failures cost the same storage as successes while looking worthless.

What gets thrown away is the only record of the boundary between working and not working. A success-only dataset can train a policy and cannot evaluate one, because it contains no negative cases, no recovery behavior, and no examples of the states that precede a failure. A project that keeps the rejects has, at no additional collection cost, a failure set that later becomes a regression test.

What failure data is actually for

The uses are not interchangeable and they have very different data requirements.

• Recovery training. An episode that fails and then recovers visits states a success-only dataset never contains, and the recovery segment is the behavior worth imitating.
• Value and progress models. These need terminal and low-progress states to learn what the end of a task looks like from the wrong side.
• Failure prediction. A classifier that recognizes an episode heading wrong can trigger an intervention before the failure happens, which is often cheaper than preventing it with policy quality.
• Evaluation. A fixed set of known failures is a regression suite: the same mistake should not reappear after a retrain.
• Diagnosis. The distribution of failure modes tells you which task to collect more of, which is a planning signal rather than a training one.

Three kinds of bad episode, only one of which is a negative

A genuine task failure is an episode where the attempt was made and did not succeed. That is a negative example. A rig artifact is an episode where tracking was lost, the operator aborted, or the recording was damaged; that is a defect, not a negative, and it belongs in the reject log rather than in the training set. A scene disturbance, where someone outside the task knocked an object or reset the workspace mid-episode, is a third category that is neither.

Collapsing all three into one failure label is what makes failure data harmful. A policy trained on a mixed pile learns that the states preceding a tracking dropout are bad, when those states have nothing to do with the task, and it will avoid them in ways that look like timidity. Three fields, one outcome, one defect flag, one cause, keep the set usable.

Near-misses are the highest-value slice

The most informative part of a failure is the moment before it. An approach that was mis-aimed by two centimeters, a grasp that was marginal but held, a placement that was off by enough that the next step would have failed: those are decisions close to the boundary, and they are where a policy has the most to learn.

So keep the whole episode and mark the onset timestamp of the failure rather than cutting the segment at collection time. A segment can be derived from a full episode later, and the reverse is not possible. The same field supports failure prediction training, which needs a window of leading frames rather than the terminal one.

Collecting failures on purpose

Waiting for failures to happen naturally produces a set that reflects the rig rather than the task, so it is worth generating them deliberately and budgeting them like any other task family.

• Perturbation episodes. Nudge the object or the arm mid-task and let the policy or the operator respond.
• Hard start states, drawn from the failures that were observed rather than invented.
• Distractor episodes, where an irrelevant object is placed to make the wrong target plausible.
• Time-limited episodes, which surface the slow failure mode that a patient operator hides.
• Scripted faults, such as a temporarily reduced grip force or a shifted camera, run in a controlled block so the effect is attributable.

Labeling and storing without waste

Every kept failure needs a mode code, a recovery flag, and the onset timestamp. The mode vocabulary should be short and stable: grasp slip, misidentification, collision, drop, timeout, unreachable, and operator abort cover most manipulation work, and a long tail of one-off codes is a sign the vocabulary was not designed.

Storage is the honest constraint. Failures cost the same per episode as successes and are worth different amounts depending on use, so the decision can be made per category. Episodes kept only as evaluation negatives can be stored with a single outcome label and no further annotation. Episodes intended as training negatives need the recovery segment and the onset marked, which is annotation budget. Deciding which category each failure falls into at ingest is what keeps the retention cost proportional to the value, and it is a decision that has to be made before the storage is reclaimed, because after that the choice is gone.