Planning a dexterous manipulation dataset: hardware, tasks, and why the yield is low

Dexterous manipulation data is the most expensive kind to collect per usable hour, and the reason is structural rather than a matter of effort. More degrees of freedom means more ways to fail, and the thing that makes the task interesting — contact between fingers and object — is the thing cameras see least well. That does not make it a bad project. It makes it a project where the expensive mistakes are made at hardware selection time and discovered at annotation time.

The hardware bar

Multi-fingered hands span a wide range. At the low end are sixteen-degree-of-freedom designs built from hobby servos, such as the LEAP Hand, where four fingers with four joints each are enough for in-hand reorientation and a broken finger does not end the project. At the high end are research hands with around two dozen joints, joint torque sensing, and repair times measured in weeks. The low-cost tier is the pragmatic starting point for data collection, and keeping a spare hand on the shelf changes the schedule more than any specification does.

Whichever hand you choose, check three things before buying: whether it reports per-joint position and current at a usable rate, whether individual fingers can be replaced without returning the hand, and whether the control interface lets you command a finger position without a long round trip. The last one decides how responsive teleoperation feels, and a hand that feels laggy will not produce good demonstrations.

Teleoperation is the bottleneck, not the hand

A human hand has more degrees of freedom than any affordable robot hand, so the operator's motion has to be reduced to the hand's kinematics, and every mapping loses something. Motion-capture gloves measure the operator's fingers directly, which is intuitive and fast to learn, but the glove and the hand rarely share a joint layout, so the mapping is an approximation that needs per-operator calibration and drifts as the glove shifts on the hand over a session.

Leader-follower finger rigs, where the operator manipulates a small instrumented hand that drives the robot hand, avoid the glove-fitting problem but put a mechanism between the operator and the task, and they are usually custom-built. VR controllers with finger tracking are the cheapest and least precise option, adequate for power grasps and poor for fine pinch. Test your task list with the teleoperation method before committing to the hand.

Why the yield is low

The reject rate on dexterous data is higher than on parallel-jaw data for reasons that cannot be engineered away.

• The operator cannot see the hand well. The fingers occlude the object and the object occludes the fingers from almost every camera angle that also shows the task, so the most informative moment is the least visible one.
• Contact is invisible. Where the fingers touch, how hard, and whether the contact is stable are not in the RGB image, and without tactile sensing the contact state has to be inferred from motion, which is unreliable exactly when it matters.
• Reset cost is high. In-hand objects scatter when a grasp fails, and a small object that lands under the table costs a minute, so a task that fails often loses most of its time to resets rather than to demonstrations.
• Success rates are lower than with a two-finger gripper, so more episodes are needed per usable demonstration, and the lower success rate and the higher reject rate compound rather than adding.

Task selection

Choose tasks that require dexterity. If a two-finger gripper can do it, the extra cost buys nothing and the resulting data is harder to use than parallel-jaw data because the action space is larger without being more informative. Good candidates are in-hand reorientation, finger gaiting to move an object within the hand, fine pinch of a thin or flat object, tool use where the grip changes mid-task, and insertion with tight clearance.

Avoid tasks whose difficulty is perception rather than manipulation. A transparent object on a cluttered table is hard for the cameras, not for the hand, and it will show up as a defect in every episode while teaching nothing about dexterity. Start with one hand model and one task family, because every additional family multiplies the reset design, the object set, and the annotation work.

What to record and what to check

Record per-joint finger positions and commands, the arm state, every camera view, and the tactile stream if the hand has one, with the tactile stream synchronized explicitly since it usually runs at a different rate from both the cameras and the arm. Check the calibration of each finger before every session, because a tendon-driven finger that has stretched will not return to the commanded position and the error is small enough to miss until it has contaminated a batch. Expect a long bring-up: the schedule should show the hand arriving, being calibrated, being teleoperated reliably, and only then producing episodes.