Scaling robot data collection across sites without splitting the dataset in two

Two kinds of difference, treated differently

Adding sites introduces two kinds of variation and they need opposite handling. Deliberate variation is the reason to add a site: a new room, a different lighting setup, a different object set, a different surface. Unintended variation is a defect: a different camera model, a mount that sits two centimeters lower, an exposure setting left on auto, a firmware version nobody pinned.

The characteristic failure of a multi-site collection is that the second kind gets recorded as the first. The dataset looks more diverse than it is, the policy learns a domain shift nobody can name, and the error is attributed to the environment when it is actually a configuration difference. The defense is to write down, per site, every value that is supposed to be identical, and to check them on a schedule.

The standardization list

Everything on this list is fixed across sites and recorded in a per-site configuration file. Everything not on the list is allowed to vary, and the variation is deliberate.

• Arm model, gripper model, and mounting geometry of the cameras relative to the workspace.
• Camera models, resolution, frame rate, and the exposure and white balance mode, with the locked values recorded.
• Control rate, action definition, and the episode boundary rules.
• Object sets, sourced from one purchase so that the physical items are identical rather than merely similar.
• Instruction templates, taxonomy version, and the session protocol.
• Calibration procedure and the record format, so two sites produce comparable calibration files.

A reference task at every site

The way to detect unintended variation before it contaminates a month of episodes is a reference task: a short, scripted or teleoperated sequence with fixed objects, fixed poses, and fixed lighting, run at every site on a schedule, with a few dozen episodes each time. Because the task is fixed, any change in the resulting numbers is a change in the rig or the environment.

Compare distributions rather than single values: episode duration, joint velocity profile, per-joint tracking error, image brightness and color histogram, and the calibration reprojection error from the session. A site whose reference distribution drifts away from the others is detectable within a week, and the record of when it started drifting is what lets you decide which episodes to trust. Keep the reference episodes in their own partition and never train on them.

Is the site difference noise or signal

Once sites are running, the question that matters is whether the difference between them helps or hurts. The measurement is leave-one-site-out: train on every site except one, evaluate on the held-out site, and repeat. If performance collapses on the unseen site, the site difference is a domain shift the policy has not learned, and either more cross-site data or explicit randomization is needed.

If performance holds, the sites are close enough that the variation is not the binding constraint, and the extra environments are adding diversity without cost. The decision that follows is about intent: if the goal is a policy that works anywhere, keep the site variation and balance the episode counts across sites so one environment does not dominate. If the goal is one specific deployment site, mixing in other environments dilutes the distribution the policy needs, and the other sites should be treated as separate datasets.

Metadata and per-site quality metrics

Every episode should carry the site identifier, room identifier, operator, hardware serial numbers, firmware and software versions, calibration identifier, and the hash of the configuration file it was recorded under. That set is what makes a later question answerable: whether a policy regression came from one site, one firmware version, or one calibration batch.

Aggregate the quality metrics per site on a dashboard: yield after ingest checks, the distribution of reject reasons, label agreement, and the reference task numbers. A site whose yield drops is one of three things, and the reject reasons usually say which. A hardware fault shows up as tracking loss or dropped frames. A training problem shows up as rejections clustered on the newest operators. A guideline problem shows up as rejections clustered on one task.

When to pool and when to keep sites apart

Pool the sites into one dataset when the taxonomy version, action definition, calibration model, and reference task are shared, because then the only difference between sites is the environment, which is exactly the variation worth having. Keep them separate when the action space differs, which happens the moment a gripper or an arm model changes, or when the calibration model differs, since an episode whose spatial transforms come from a different procedure cannot be mixed with the rest.

Pooling is reversible as long as the site identifier is in the metadata, and it is not reversible if it is not. That asymmetry is the argument for recording the identifier even when the plan is a single pooled dataset: the cost of the field is nothing and the cost of its absence is a collection that can never be split.