Choosing a format for robot data: RLDS, HDF5, or LeRobot-style datasets
Three formats dominate robot learning, and each one is optimized for a different bottleneck: streaming, inspection, or sharing. What to store per step.
The format decision is usually made by accident. Whatever the collection script happened to write becomes the dataset, and it stays that way until the day it becomes a problem. It is worth deciding deliberately, because changing the format later means rewriting every loader and re-encoding every image in the collection.
Keep the raw capture and convert on top of it
Whatever the robot records natively — a ROS bag, an MCAP file, or per-episode files written by the collection script — is the archive, so keep it. Conversions lose information, and the information they lose is generally what you need six months later. The training format is a derived artifact, which makes the converter part of the dataset and means it needs a version number, because an episode converted by two different versions of the converter is two different episodes.
RLDS: built for streaming
RLDS packages episodes as TFRecords through the TensorFlow Datasets machinery, with each episode split into steps carrying observation, action, reward, discount, and first-step and last-step flags. It is the format Open X-Embodiment standardized on, which is a genuine advantage if you want to mix your data with public datasets. Sharded files stream efficiently into a training pipeline, the schema is fixed and widely understood, and combining sources is straightforward once everything conforms to it.
The costs are real. TFRecord is not randomly accessible, so you cannot jump to episode four thousand without reading up to it, and debugging means reading shards rather than opening a file. Writing a dataset requires a builder script, and images stored as encoded bytes add a decode step that has to behave identically in training and evaluation, which it usually does not on the first attempt.
HDF5: built for inspection
One file per episode, arrays under named groups, readable from Python or from a viewer. This is the format the ALOHA work used, and it is the easiest to debug: open a file, plot a joint, look at the images, attach metadata per episode. When something is wrong with an episode, this is the format that tells you what.
It does not shard, so a large collection becomes a directory holding tens of thousands of files and a filesystem that starts to struggle. Images stored as raw arrays make files enormous, and concurrent writes to one file are a bad idea. It scales comfortably to thousands of episodes and not comfortably to millions.
LeRobot-style: built for sharing
Episode metadata in Parquet tables with frames in encoded video files, distributed through a dataset hub. The appeal is size and portability: storing frames as video is dramatically smaller than storing them as individual images, and the metadata stays queryable with ordinary table tools.
Retrieving an exact frame requires seeking inside the video, which is slower than reading an array and can be imprecise depending on the encoding, and decoding during training costs CPU or GPU. Because frames are compressed, the format is lossy in a way HDF5 arrays are not. That is acceptable for most training and not for tasks that depend on fine texture or exact pixel values, which makes this the right choice when the primary goal is publishing a dataset.
What to store for every step
The format matters less than what goes into it, and the fields people regret leaving out are consistent across projects.
- Timestamp and the resampled control index. Keep both, because the original timestamp and the resampled one are not the same thing, and only the original lets you redo the alignment.
- Joint positions, commanded actions, and gripper state, with the command and the achieved state stored separately, because their difference is the only clean way to diagnose tracking error later.
- End-effector pose, whether measured or derived. Cheap to store, tedious to reconstruct, and the reconstruction is never quite the same as the original.
- Images from every camera under distinct keys, with the camera identifier in the metadata rather than in the key name, so that a camera change does not require renaming every field.
- Episode metadata: task, instruction, operator, date, scene, object set, success label, and calibration identifier. This is what makes a collection filterable later, and it is nearly impossible to add retroactively.