Embodied AI data

What counts as embodied AI data

Embodied AI covers robots, autonomous vehicles, drones, and software agents that control something in the physical world. What separates its data from ordinary training data is that every sample carries a record of action and consequence: what the system perceived, what it did, and what happened next.

A caption on an image says what is in the picture. An embodied sample says where the arm was, how it moved, whether it made contact, and whether the grasp held. That extra structure is what makes the data expensive and what makes it useful.

The modalities a project usually combines

• Egocentric video — first-person footage from a head-mounted or wrist-mounted camera, showing hands and objects from the actor's own viewpoint.
• Teleoperation trajectories — a human drives the robot and the resulting joint positions, gripper states and timestamps are recorded as a demonstration.
• Multi-view capture — the same scene from several fixed cameras, so a 3D pose can be reconstructed from the overlap.
• Motion capture — human body pose tracked at joint level, used where the target is a humanoid or where human motion is the reference.
• Tactile and force — contact pressure and grip force, needed for manipulation tasks where vision alone cannot tell whether something is slipping.
• Action and affordance annotation — labels describing what can be done with an object, not merely what the object is.
• Failure data — unsuccessful attempts, which are usually more informative than successes and much harder to collect deliberately.
• Speech and instruction — spoken commands paired with the actions they produced, for language-conditioned control.

Why it is expensive

Cost is driven by time-on-task rather than by volume. An hour of ordinary speech data is one hour of a speaker talking. An hour of teleoperation is one hour of a trained operator standing at a rig, plus the rig, plus the objects being manipulated, plus the resets between attempts.

Two consequences follow. First, the same scene shot from a different angle is cheap; the same scene with a different task is not. Second, projects should be scoped around the number of distinct tasks and object types, not around hours, because that is what actually determines how long the collection takes.

The generalisation problem

A dataset recorded in one kitchen produces a model that works in that kitchen. The variability that matters is not the number of repetitions but the spread of conditions: different lighting, different surface materials, different object shapes within the same category, different clutter levels, different camera mounting positions.

This is why embodied collection is usually specified as a matrix rather than a total. Thirty hours spread across six environments and forty object categories teaches more than thirty hours in one room, even though the file count is identical.

What to specify

• Task list — the concrete actions, written as verb plus object, not as a category name.
• Environment spread — how many distinct locations, and what varies between them.
• Object inventory — the physical items needed, including how many of each and whether they must be sourced by the producer.
• Sensor configuration — camera model, mounting, frame rate, whether depth is captured, whether force or tactile sensing is included.
• Robot platform — the exact model, since joint conventions and action spaces differ between them.
• Calibration record — intrinsic and extrinsic camera parameters, without which multi-view data cannot be fused.
• Success criteria — what counts as a completed attempt, and whether failed attempts are retained.

Consent and privacy in physical capture

Egocentric and multi-view capture routinely records people who did not agree to be recorded — bystanders, family members, colleagues in the background. A dataset can be technically clean and still carry a consent problem that surfaces much later.

The practical controls are to record in controlled spaces, to define a blurring or exclusion rule for bystanders and apply it before delivery, and to document what that rule was. Where a person appears deliberately, ordinary consent requirements apply as they would for voice.

How we source it

We do not hold embodied datasets in stock. A project starts from your task list and sensor configuration, and we identify a facility or field team that has the platform, the space and the operator skill to produce it. Where the platform is unusual, we will say so early rather than after a quote.

For a project that combines manipulation data with speech, both are produced under one specification and delivered together, which is usually cheaper than running them separately.

Related pages

• First-person view data — egocentric capture, and how it is annotated.
• Autonomous driving data — the road-going case of embodied collection.
• Speech command datasets — the language side of language-conditioned control.
• Choosing a data provider — what to check before commissioning collection.
• Compliance and consent — how bystander and speaker consent is documented.