Sim-to-Real

Sim-to-real RL trains a simulated version of a robot or embodied system before deployment in the real world. The motivation is straightforward: simulated data is cheap, fast, and safe. An agent can practice dangerous or expensive behavior in simulation before anyone risks real hardware, real money, or real people.

The hard part is transfer. A policy trained in simulation may depend on physics, sensors, noise, or dynamics that do not match reality. If the simulator is the wrong world, the agent can optimize actions that break in the real one.

Sim-to-real reinforcement learning pipeline
The overview of Sim-to-Real issue causes: The Action Gap, The Reward Gap, the Next State Gap, and the Observation Gap. [Source]
Sim-to-Real Robots IRL

Sim-to-real

Where does the simulator differ?

Pick a transfer gap. Sim-to-real RL is powerful because simulated data is cheap and safe, but policies can break when reality differs.

Risk

A policy that balances perfectly in simulation may slip, bounce, or collide on hardware.

Mitigation

Randomize dynamics and validate on real rollouts.

The recurring model-based challenges are model bias, compounding error, and uncertainty.

Sim-to-real techniques

Where should we intervene?

Select a transfer gap to see the technique families used to reduce it.

Techniques

Transition gap

Make simulated dynamics transfer to real-world state changes.

Domain RandomizationDomain AdaptationGrounding MethodsDistributionally Robust RLLLM-Enhanced Approaches
A sim-to-real method usually targets one of four gaps: observation, action, transition, or reward. [Source]

Good model-based RL is uncertainty-aware. It should distinguish between "I predict the ball will bounce like this" and "I have no idea; I have never seen this state before." Practical tools include short imagined rollouts, latent planning, regularization that keeps plans realistic, simulator randomization, and frequent refreshes from real data.

Checkpoint

Why can a policy trained in simulation fail after transfer to the real world?

←PreviousMuZeroModel-Based RL and World ModelsNext→Temporal AbstractionHierarchical RL