The Sim-to-Real Gap Is a People Problem, Not a Physics Problem

After working through sim-to-real transfer on multiple robot learning projects, I've come to a conclusion that surprises most people: the gap isn't primarily about physics fidelity. It's about the assumptions baked in during design that nobody writes down.

Physics engines have gotten remarkably good. Mujoco, IsaacSim, and others can approximate contact dynamics, friction, and deformable objects well enough for most manipulation tasks. The sim-to-real gap I've encountered most often isn't "the simulation was wrong" — it's "we optimized for the wrong thing in simulation and didn't know it until we put the robot in front of real objects."

The most common failure mode I've seen is overfitting to the simulation's observation space. A policy that achieves 95% in sim often does so by exploiting subtle regularities — perfect depth estimates, noiseless joint encoders, textures that never vary — that simply don't exist in the real world. The policy isn't learning the task; it's learning the simulator. I've had to rebuild pipelines specifically to prevent this, adding noise profiles, randomizing sensor models, and deliberately breaking the "clean" data that simulation naturally produces.

The second failure mode is harder to fix: misaligned success criteria. If your simulated reward says "task complete" at a slightly different threshold than what actually matters in the field, you can train for weeks and still ship something that looks wrong to a human operator. Getting this alignment right requires field experience — you have to know what good looks like in the real environment before you can encode it in simulation.

The fix isn't a better physics engine. It's spending more time in the problem definition phase, testing your sim assumptions against real data early, and treating simulation accuracy as something you measure and validate — not assume.