Designing a task taxonomy for a robot dataset

One node, one success criterion

The test for whether something deserves its own taxonomy node is simple: could two people disagree about whether an episode succeeded at it? If they could, either the success criterion is missing or the node is really two tasks. A taxonomy node is an evaluation unit before it is a label, and a collection whose nodes do not map to checkable outcomes cannot be evaluated per task.

Three levels are enough: the skill, the task, and the task family. A skill is one verb against one object class, a task is a skill plus a target and a constraint, and a family groups tasks that share a success criterion and a motion pattern. Deeper hierarchies look tidy on paper and collapse in practice, because annotators cannot hold more than three levels consistently and reviewers cannot audit them.

Verb-object, with a controlled verb list

Name nodes as a canonical verb plus an object class, and keep the verb list short and closed. Pick, place, insert, stack, open, close, pour, fold, wipe, and hand over cover most tabletop work. The reason to close the list is that synonyms are the main source of accidental duplication: grab, grasp, and pick up will be used interchangeably by different annotators unless one is declared canonical, and the resulting dataset has three tasks where it should have one.

Keep the object class in a separate field rather than inside the name. A node called pick_can cannot accept a bottle without a new node, while a node called pick with object set to can or bottle generalizes by adding a value. The noun side deserves the same discipline as the verb side: a controlled list of object classes, with size, material, color, and instance identifier stored as facets rather than folded into the class name.

Facets belong in fields, not in the name

Every facet that lives inside a name is a facet nobody can filter on later, and names are the most expensive thing in a dataset to change because they are matched by strings everywhere downstream.

• Object class and instance, with physical attributes kept separate.
• Target: the container, surface, or location the object goes to, identified by name rather than by coordinates so the record survives a scene change.
• Spatial relation: in, on, under, beside, aligned with.
• Constraint: upright, within the marked area, without touching the rim.
• End state: the state change the task is supposed to produce, such as standing versus flat.
• Distractors: whether other objects were present, and whether they were relevant.

Generate the instructions from the taxonomy

If the taxonomy is structured, the instruction text is a template filled from it: one template per verb with slots for object, target, and constraint. Writing instructions by hand alongside the taxonomy guarantees they drift apart, because two annotators describing the same node will choose different words, and a language-conditioned policy learns to read that phrasing difference as a task difference.

Store the slot values, not only the rendered sentence, so that instructions can be regenerated when the phrasing is revised. Keep a small set of paraphrase variants per node if the policy is meant to be robust to phrasing, with the node identifier as the ground truth and the paraphrase as a secondary field. The taxonomy is the label; the sentence is a rendering of it.

Splits follow families, not episodes

A random split by episode leaks, because the same scene, the same object, and often the same initial pose appear on both sides of the split. Splitting by task family is the version that measures generalization, and a taxonomy makes it possible to state the split precisely: hold out one family, one object class, and one target, and report results on each.

This is also a design constraint on the collection itself. A taxonomy where every node has a single instance cannot support a held-out split, because holding out the instance leaves nothing to train on. Plan for at least two instances per node if generalization is part of the claim.

Version it and stop renaming

Node identifiers are append-only. A node that is renamed keeps its identifier, a node whose definition changes gets a new identifier or a versioned definition, and numbers are never reused, because episode records elsewhere in the pipeline point at them. Ship the taxonomy as a machine-readable file with the dataset and write its version into every episode record.

Failure reasons are a separate axis, not taxonomy nodes. Mixing them produces nodes like pick_can_failed, which is a task and an outcome in one field and cannot be filtered independently. Keeping outcome, failure mode, and task identity in three fields is what lets the same collection answer a question nobody has asked yet.