Designing a pilot batch before you scale a robot data collection
How large a pilot should be, which task variants to include, and the thresholds that tell you whether to scale up or fix the pipeline first.
A pilot is not a demonstration. Its job is to find the ways the pipeline breaks while fixing them is still cheap, and a pilot that only shows the rig works has tested the least interesting thing about the project. The value comes from producing a distribution instead of an anecdote: one session on one task tells you almost nothing, while a few hundred episodes across a few task families tells you what the yield is, what the labeling disagreement rate is, and whether a policy can learn from the data at all.
Size it to answer one question per dimension
A pilot of a few dozen episodes is too small, because a single bad operator day dominates the result and every number looks like noise. A pilot that matches the full collection is not a pilot, it is the project, and it removes the option to change course.
A workable shape is a few hundred episodes split across three to five task families, two operators, and two scenes or object sets. That is enough to see the spread of episode length and yield per task and to compare operators against each other. Include at least one task you expect to be hard, because a pilot built from easy picks reports a success rate the real collection will never reproduce.
Choose the variants deliberately
Variation is the axis most pilots get wrong in one of two directions: everything is fixed, so the result is uninformative, or everything varies, so nothing is attributable.
- Object pose. Full randomization from the start, or a fixed set of start poses? Fixed poses collect faster but hide whether the policy can handle variation at all.
- Lighting. Use at least two conditions, including the one you expect at deployment. Overhead-only lighting produces policies that fail under mixed light, and the pilot is where that shows up cheaply.
- Camera viewpoint. Hold it fixed during the pilot. If the camera moves between sessions, a performance drop could be the policy, the calibration, or the lighting, and you cannot tell which.
- Distractors. Include a handful of episodes with irrelevant objects in frame. A policy that only ever sees the target object learns to assume it is the only thing present.
- Instruction phrasing. Write two or three phrasings for the same task and check whether the annotations stay consistent across them.
Hold something back from day one
Reserve one scene, one object set, or one lighting condition that never enters training. If the pilot only evaluates on data drawn from the same distribution it trained on, it will report success and tell you nothing. The held-out set does not need to be large: a few dozen episodes is enough to reveal a collapse when a policy has overfit to the pilot scenes, and it costs almost nothing to set aside at the start and a great deal to reconstruct later.
Set the thresholds before you look at the numbers
Write the criteria down before the pilot runs. Every result looks acceptable once the money has been spent, and thresholds invented afterwards are not thresholds.
- Yield after quality filtering. What fraction of recorded episodes survives the checks. If the yield is low and the rejects cluster around one operator or one task, fix the pipeline before scaling it.
- Label agreement. Double-label a slice of the pilot and measure how often two annotators agree on success and on task boundaries. Low agreement means the guideline is not finished, and scaling multiplies that error.
- Baseline learning check. Train a small imitation policy on the pilot and see whether it can fit the demonstrations. A model that cannot reproduce its own training episodes usually means the action-observation alignment or the timestamps are wrong, not that the model is too small.
- Cost per usable episode. Divide the real cost by the episodes that survived, not by the episodes recorded. That is the number to compare against the alternative of buying data instead of collecting it.
Write down the pilot while it is fresh
Record the rig configuration, calibration values, control rate, camera settings, and the exact instruction templates. Scaling with a slightly different setup produces a dataset with two incompatible halves, and the difference is usually found during training rather than during collection. The most valuable output of a pilot is normally the list of things that were underspecified, and that list is the specification for the real collection.