What is Robot Manipulation Data?
Robot Manipulation Data — episodes of a robot arm moving objects, recorded together with the commands that produced the motion.
A manipulation episode is a time series, not a clip: camera frames, joint positions, gripper state and the action command, all aligned on one clock. The control rate is the first number that matters. ALOHA-style bimanual rigs record at 50 Hz, while a large multi-lab set like BridgeData V2 runs at 5 Hz, and a policy trained at one rate does not transfer cleanly to the other.
The common formats are HDF5 (one group per episode, the layout ALOHA and ACT use), RLDS, which is how Open X-Embodiment is distributed, and the LeRobot format, which keeps low-dimensional state and action in Parquet files and camera streams as MP4. How many demonstrations a task needs is a scoping question: the ACT work found about 50 per task sufficient for fine bimanual skills, while RT-1 collected over 130,000 episodes across more than 700 tasks — roughly 150 to 200 per task.
What makes this data hard is that demonstrations are only as useful as their coverage. Fifty episodes of the same grasp teach one grasp; the value is in object variety, lighting, camera placement and the failure cases most producers discard. Sets of this kind are built to order — the task list, platform and sensor configuration are specified first, and the episode count follows from them.
Related terms
-
Teleoperation
Teleoperation — a human drives the robot directly, and the robot's motion is recorded as a training demonstration.
-
Vision-Language-Action Model
Vision-Language-Action Model — a model that takes camera images and a written or spoken instruction and outputs robot actions directly.
-
Robot Data Annotation
Robot Data Annotation — attaching labels to robot episodes: what the task was, whether it succeeded, and when key events happened.
Keep reading
-
Full glossary
Every term we explain, in one list.
-
How to buy training data
Where these terms actually show up, and which ones change a quote.
-
Help center
How a project runs, from specification to delivery.
Submit a sourcing request
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline — not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.