Closing the sim-to-real gap in practice: what to randomize and how much real data to add

Sim-to-real transfer is usually described as a gap in the physics. In practice it is a measurement problem: the gap is the difference between what you assumed about the real system and what it actually does. The first useful step is not turning up randomization, it is writing down the assumptions and checking the ones you can check.

Measure the real system before simulating it

Latency first. Command a step change and measure how long until the joint actually moves; the total delay includes the controller, the link, and the actuator response, and it is usually larger than people assume. Then dynamics: the friction of the objects you actually use, their mass, and how the gripper's commanded force is realized. Friction is the one people skip, and an inclined-plane test with the real object on the real table gives a usable number in a few minutes.

Then sensing. Record a static scene and measure depth noise and frame-to-frame RGB variation at several distances, because real camera noise is not Gaussian and it grows with distance. The sanity check that catches most problems: run a scripted motion in both simulation and reality, send identical joint commands, and compare the resulting paths. If they diverge on a motion with no learning and no perception involved, fix that before training anything.

What to randomize, and how wide

Randomization is a range, not a switch, and the width of the range is the decision that matters.

• Visual: lighting direction and intensity, color temperature, table and object textures, camera pose and intrinsics jitter, background clutter, and a few distractor objects. These are cheap to vary and cover the variation you cannot enumerate in a real setup.
• Dynamics: mass and inertia of the manipulated objects, surface friction, actuator gains, and control latency. Randomize around the values you measured, not across the whole space of physical possibility.
• Sensor: depth noise as a function of distance, motion blur, and exposure variation. If the real camera runs at a locked exposure, do not randomize exposure to extremes in simulation, because you will train the policy to handle conditions that never occur.
• Over-randomization has a specific failure mode: a policy that is robust to everything and competent at nothing, because it hedges. If the policy performs poorly in simulation, the range is too wide. Narrow it until the policy is competent, then widen gradually.

Where real data goes in the plan

There are three common shapes. Train in simulation and fine-tune on a small set of real episodes, where tens to a few hundred is often enough when the simulation is close and the task is single. Use real data to calibrate the simulation through system identification and then train mostly in simulation, which spends the fewest real hours and the most engineering time up front. Or co-train on mixed simulated and real batches so the policy sees both distributions throughout, which tends to be the most robust and the most expensive in infrastructure.

Whichever shape you choose, keep real collection running while simulation work continues, because the real episodes are also the evaluation set. A policy that has never been tested on hardware is a hypothesis, not a result.

Evaluate on the thing you actually care about

Test on held-out real scenes and objects, with enough trials to separate a real difference from luck. A success rate measured over a handful of trials cannot distinguish a policy that succeeds sixty percent of the time from one that succeeds eighty percent of the time, and that difference is exactly what a transfer project exists to measure. Watch the rollouts rather than only the success metric, because policies that succeed in simulation by exploiting the contact model, pushing objects through corners or gripping with impossible force, fail immediately on hardware.

Report the failure modes alongside the number. A policy that drops the object is a different problem from one that never finds it, and the two have different fixes. A single success rate hides which of them you have.

Where simulation still loses

Deformables, cloth, rope, granular material, and anything where the contact area changes continuously are still weak in simulation, and randomization does not repair a contact model that is wrong in kind rather than in magnitude; adding noise to a fundamentally incorrect model produces a policy that is robustly wrong. Tactile sensing is hard to simulate faithfully, so the contact-rich portions of a task need real data. The practical rule is to use simulation where the physics is rigid-body and perception is the hard part, and real data where the contact is the hard part.