Evaluating a robot policy before deployment: eval sets, metrics, and regression tests

The evaluation set is a data collection project

A policy evaluation is only as good as the conditions it runs on, and those conditions have to be built with the same discipline as the training data. The useful form is a manifest: a written list of initial states, each one reachable and reproducible, with the object poses, the scene, the lighting condition, and any randomization seed recorded alongside.

The manifest carries a version number and stops changing once a result has been recorded against it. If the eval set moves between checkpoints, the comparison between them means nothing, and the failure is quiet because both sides still look like success rates. Collect the eval episodes deliberately, hold back a scene, an object set, and a lighting condition that never appear in training, and treat that holdout as a commitment rather than a preference.

Twenty trials cannot separate two policies

Success is a binomial outcome, so the uncertainty around it depends on how many trials produced it. Twenty trials at sixty percent success give an interval wide enough to overlap eighty percent, which means the evaluation cannot tell a working policy from a better one. Separating sixty from eighty percent at conventional confidence takes on the order of eighty to a hundred trials per condition.

That number is per condition, not per policy. If the task has four object classes and two lighting conditions, a defensible evaluation is several hundred rollouts, which is why evaluation hardware time belongs in the project budget as a first-class cost rather than something that happens at the end. Report intervals rather than bare percentages, and report the number of trials next to every result so nobody reads a twenty-trial run as a finding.

Metrics that predict deployment

Success rate alone hides the difference between a policy that fails cleanly and one that fails dangerously, and it hides how much supervision the system needs to keep running.

• Partial progress. Stage completion or distance-to-goal at the end of a rollout. A policy that gets the object to the bin ninety percent of the way is a different starting point for improvement than one that never finds it.
• Intervention rate. Human takeovers per hour of operation. This is usually the number that decides whether a deployment is viable, because it converts directly into staffing.
• Time to completion and its spread. A policy that is fast on average and occasionally takes three times as long behaves worse in a production line than its mean suggests.
• Recovery rate. How often the policy recovers after a perturbation or a failed grasp, measured on deliberately perturbed rollouts.
• Severity of failure. A dropped object, a collision, and a stalled episode have very different costs even when they are all labeled failure.

Report per condition, not just in aggregate

An average over conditions hides the tail, and the tail is what deployment tests. A policy at eighty percent overall that sits at thirty percent on one object class will be described in the field as unreliable, and the aggregate number will have predicted the opposite.

Break results down by object class, lighting condition, start pose region, and task variant, then look for the worst cell rather than the mean. Include a baseline in the same table, either a scripted controller, a teleoperated human, or the previous policy version, because a success rate with nothing to compare against cannot support a decision. Watch the rollouts themselves rather than only the numbers: a policy that succeeds by pushing an object through a corner is exploiting something that will not hold on hardware.

Regression testing across checkpoints

Once a policy has a baseline, every later checkpoint should run the same frozen suite, and the result should be a diff rather than a new headline number. The specific thing to look for is a regression on conditions that the latest change was not supposed to touch: a fine-tune on one task family that quietly costs five points on another is the most common way a model gets worse while the dashboard improves.

Seed variance is the other thing to keep visible. Run the suite with several seeds and report the spread, because a checkpoint that is better by two points with a ten-point seed spread has not been shown to be better at all. A small suite run on every checkpoint beats a large suite run once, and the small suite should always include the conditions the policy has failed before.

What a deployment gate should contain

Write the gate as a set of conditions that must hold, agreed before the final checkpoint is trained: a minimum number of trials per condition, a floor on the worst condition rather than the average, an intervention rate ceiling, and a maximum on failures of the severe category. Name the deployment environment in the gate document, because a policy evaluated in a bright lab and deployed under mixed light has been tested against a different problem.

The last piece is a supervised rollout plan. Deployment should start with a human present and the intervention rate recorded, because production is the only evaluation that uses the real distribution. The gate decides whether to start that phase, not whether the policy is finished.