Physical AI Data

What the term covers

Physical AI is a newer label for an old requirement: training data in which the system perceives a physical situation, takes an action, and records what happened. Robotics and autonomous driving are the established cases; drones, industrial inspection, and wearable-captured human activity are the expanding ones.

The label matters less than what it implies for data. Every sample needs three things joined together — the sensor record, the action or state, and the outcome. Data that has only the sensor record, however high its quality, is not physical AI training data.

Machine-collected and human-collected

There are two families of collection, and the budget conversation goes better once the buyer picks one.

Machine-collected data comes off a platform: teleoperation episodes, autonomous runs, fleet logs. It carries true joint states and precise action records, and it costs operator and rig time. It is the right choice when the model has to control that platform.

Human-collected data comes off people: egocentric video from head or wrist cameras, motion capture, hand-pose tracking, wearables. It is cheaper per hour, scales with the number of participants rather than the number of robots, and is the raw material for imitation learning and for world models that predict how a scene evolves. It is the right choice when the target is general manipulation understanding rather than one machine.

Most serious programs end up buying both, because human data supplies breadth and machine data supplies control fidelity. Buying them under one specification is cheaper than running two projects, because the scene definitions and the annotation schema can be shared.

What drives cost and how long it takes

Sensor configuration is the first cost driver: more cameras mean more synchronization, more calibration, and more storage, and depth or tactile channels add their own processing. Annotation of interaction is the second — labeling what a hand is doing to an object is slower than labeling what the object is, and contact events, affordances, and failure moments are the labels that carry the value.

Human-collected data has a cost that machine data does not: the people in the frame. Participants have to be recruited, briefed, and consented, and bystanders who wander into egocentric footage have to be excluded or blurred by a rule that is documented and applied consistently.

A pilot is possible within weeks once the sensor setup is fixed. Production then scales with participants and scene count rather than with studio time, so a human-collected project of a few hundred hours can finish faster than an equivalent robot project — recruitment is the pacing item.

What you receive and what to settle first

Deliverables follow the family of data. Machine-collected projects deliver episodes with aligned streams and per-session calibration records. Human-collected projects deliver episodes with camera calibration where relevant, participant consent documentation, and the blurring or exclusion rules that were applied. Both come with the annotation schema and a QA report covering what was checked and what was rejected.

Before production, the buyer settles four things: which family the project needs, the sensor or capture configuration, the task and object taxonomy the annotation will follow, and the rights position for the people recorded. The last one is the one buyers most often leave late, and the one that cannot be fixed after capture.

One boundary to state early, because physical AI work runs close to it: we do not take on medical or clinical data. Surgical and clinical capture is out of scope here even when the platform is a robot, and recorded telephone calls are declined for the same reason they are everywhere else — the other party never consented.