Calibration records are part of the dataset, not internal housekeeping

What a dataset without calibration cannot do

Episodes without calibration values still look fine. The video plays, the joint arrays load, and a policy can be trained on the observation and action streams directly, which is why the omission survives the first training run and is discovered later.

What is lost is everything spatial. Without camera intrinsics, depth cannot be reprojected onto color. Without extrinsics, a detected object pose cannot be transformed into the arm frame. Without the kinematic offsets, a trajectory cannot be reconstructed in world coordinates, a relative end-effector action cannot be re-derived from joint positions, two streams whose sync was wrong cannot be re-aligned, and two rigs cannot be combined into one coordinate frame. Every one of those is a task someone will want in six months, and none of them can be recovered from the video.

Camera intrinsics: the fields that matter

The numbers are the easy part. The fields that make a record usable are the ones that say what the numbers apply to.

• The resolution the calibration was computed at. Intrinsics are resolution-specific, and rescaling them to a different resolution has to move the principal point as well as the focal lengths.
• Focal lengths and principal point, in pixels, with the convention stated, since some tools report a normalized focal length instead.
• Distortion model name and coefficients. The model name matters as much as the values, because a fisheye model and a radial-tangential model are not interchangeable and applying one as the other produces a plausible-looking wrong projection.
• The target used, its physical size, the number of views captured, and the tool and version that solved it.
• Reprojection error in pixels, which is the acceptance metric. Sub-pixel error is achievable with a good printed target; a record above a pixel is a warning that the calibration was not well constrained.

Extrinsics need frames, not just matrices

A transform without a stated parent frame and child frame is not a transform, it is four rows of numbers. Every extrinsic record should name the two frames, give the translation and rotation, and state the convention, including the direction, because camera-to-base and base-to-camera differ by an inverse and the mistake produces a mirrored result that looks almost right.

The set worth recording: camera to robot base for each camera, camera to camera for any pair used together, gripper or tool center point to the wrist flange, and table to base if the table is the reference for object poses. Keep the nominal design geometry and the measured transform as separate fields, because the difference between them is a diagnostic: a measured value that has drifted away from the design value means something moved.

Time offsets and the non-camera sensors

Each stream has its own clock and they do not agree. Record a per-stream offset with a sign convention and the method used to measure it, whether that was a visible flash, a marker crossing a known point, or an audio event captured by two devices. Offsets drift over a long session, so they are per-session values rather than per-rig constants.

The arm side has its own record. Joint zero offsets and direction signs, the tool center point, the gripper width reading at fully open and fully closed, force and torque sensor bias with the gravity compensation parameters, inertial unit biases, and a tactile baseline per taxel if the hand has tactile sensing. These are the values that decide whether two episodes from the same rig are actually comparable, and they are the ones most often missing from delivered data.

How to package and reference it

One record per calibration event, with a stable identifier that never changes once assigned, a timestamp, the trigger reason, and the tool versions. Store it as a version-controlled text file alongside the dataset, with a hash, and write the identifier into every episode recorded under it. Keep the raw calibration session as well, whether that is the target images or the recorded stream, so the solution can be recomputed with a better tool later.

Then make the pipeline enforce it. A dataset acceptance check should fail when an episode references a calibration identifier that does not resolve to a file, and it should fail when two episodes in one batch reference calibrations that differ in a way that changes the geometry. The rule to state plainly is that the calibration record is a deliverable, on the same footing as the episodes themselves.

When to re-calibrate

The triggers are worth writing into the session protocol, because a calibration that is only done at the start of a project is stale within a week. Re-calibrate after any contact with a camera mount, after a lens or camera swap, after a gripper change, after a crash, after a firmware update that alters streaming or resolution, and at a fixed interval during a long collection even when nothing appears to have changed.

Record the reason for each calibration event, not only its values. When an episode later looks wrong, the trigger log is what tells you whether it fell inside a window where something had moved and had not yet been re-measured, which is often the difference between discarding one session and discarding a month.