Choosing cameras for robot data capture: geometry first, sensor second

Start from pixels on the object

Camera choice looks like a sensor specification problem and is mostly a geometry problem. The number that decides whether a policy can learn a task is how many pixels the manipulated object occupies in the view that matters, and that number comes from object size, distance, field of view, and image width.

The arithmetic is worth doing before buying anything. A five centimeter object at half a meter, seen through a seventy degree lens with a six hundred forty pixel wide image, lands at roughly forty-five pixels across, because the view is about seventy centimeters wide at that distance and there are about nine hundred pixels per meter. A hundred and twenty degree lens cuts that by well over half, because the view is nearly two and a half times wider at the same distance. Coarse pick-and-place survives around fifty pixels; insertion or fine alignment wants several hundred, and doubling resolution or halving the distance is what buys them. A wider field of view is a cost paid in object pixels, not a feature.

How many frames the contact lasts

The decisive moment in a manipulation episode is short. A grasp can begin and finish within a tenth to three tenths of a second, which is three to nine frames at thirty frames per second and six to eighteen at sixty. If the failure you care about is a slip or a re-grasp, thirty frames per second may simply not contain it, and no amount of resolution fixes a moment that was never sampled.

Frame rate also has to relate to the control rate the policy runs at. An action chunk of sixteen steps at thirty hertz covers about half a second, and the observation stream should be sampled on a schedule that makes that chunk meaningful rather than on whatever rate the camera happens to offer. Shutter type belongs in the same conversation: a rolling-shutter color sensor skews fast lateral motion, and many RGB-D units pair a global-shutter depth sensor with a rolling-shutter color sensor, which is easy to miss on a specification sheet.

Exposure, focus, and color must be locked

Auto-exposure is the most common silent defect in robot capture. The operator body crosses a light source, the camera compensates, and every following frame has a brightness the policy has never seen in that scene. Lock exposure and gain, lock white balance, and record the locked values in the session configuration so a later session can reproduce them.

Fixed focus and fixed focal length are the same kind of decision. Autofocus hunting produces blurred frames exactly when the arm moves, which is exactly when the data matters. Color temperature deserves a note in the metadata as well: a room lit at a warm temperature and the same room lit at a daylight temperature produce visibly different histograms, and a policy trained on one may read the other as a different scene.

Depth, and the interference nobody plans for

Active depth cameras project an infrared pattern, and two of them looking at the same scene see each other pattern. The result is holes, banding, and depth errors that appear only when both cameras are on, which makes the defect look like a hardware fault in one unit. The known mitigations are to disable the emitter on all but one camera, to use different projector patterns where the hardware supports it, to time-multiplex the emitters with a sync signal, or to make the second view passive.

Depth also fails on surfaces that do not return the pattern usefully: shiny metal, clear plastic, and very dark material. That is a property of the object set, not of the camera, and it is worth checking against the actual objects before depth is specified as a required input rather than an optional one.

Bandwidth and cabling decide how many cameras you get

Image data is heavy before compression. A seven-twenty-pixel color stream at thirty frames per second is on the order of eighty megabytes per second uncompressed, and a USB controller carries a few hundred megabytes per second shared across every device attached to it. Three or four cameras on one controller is where dropped frames and enumeration failures begin, so cameras belong on separate host controllers, and a powered hub or an add-in card is safer than a bus-powered one.

Cabling causes more intermittent faults than sensors do. A cable that is loaded by the arm motion will eventually produce a camera that drops frames only in certain poses, which is the hardest kind of defect to reproduce. Strain-relieve the cable at the mount so the connector never carries the load, and keep spare cables on the shelf.

Match the capture to what the policy consumes

Decide the policy input before buying, because capture resolution above the training resolution is discarded at load time. The useful test is the object pixel count at the resolution the policy will see, not at the resolution the camera records: a rig that captures at high resolution and downsamples to a small square for training may be below the object-pixel budget even though the raw footage looks excellent.

Two conventions are worth fixing at the same time. Camera naming and channel order have to be identical across sessions and rigs, because a policy trained with the wrist view in one position fails when the order changes, and the failure looks like a policy problem rather than a data pipeline problem. And every camera needs its calibration on record, since a camera set without intrinsics and extrinsics produces episodes whose spatial content cannot be used, however good the images look.