Planning with Learned Dynamics

Once an agent has a learned model, it can use the model to imagine possible futures before acting. A delivery robot can compare a shortcut through a crowded corridor with a slower route around it. A recommender can estimate whether showing one item now will make later interactions better or worse. A game-playing agent can search several moves ahead instead of reacting only to the current position.

This is the central appeal of model-based RL: it can turn data into reusable experience. The agent does not need to execute every candidate action in the real world. It can plan, evaluate, and sometimes train on imagined rollouts generated by the learned dynamics.

The planning loop repeatedly feeds predicted states back into the model:

$$\hat{s}_{t+k+1}=f_\theta(\hat{s}_{t+k},a_{t+k})$$

A candidate action sequence can then be scored by predicted reward and final value:

$$\hat{G}_t=\sum_{k=0}^{H-1}\gamma^k\hat{r}_{t+k+1}+\gamma^H V(\hat{s}_{t+H})$$

The horizon $H$ controls how far the agent imagines. A larger $H$ can reveal delayed consequences, but it also gives errors more time to compound. If the model is a little too optimistic about a slippery turn, the first imagined step may be only slightly wrong. Ten imagined steps later, the plan may rely on a trajectory the real robot cannot follow.

Modern model-based systems often respond with a deliberately modest use of imagination. Model predictive control replans after each real step instead of committing to a long open-loop plan. MBPO-style methods use short model rollouts, often only a few steps, to create extra training data without asking the model to predict far into unfamiliar futures. Ensembles and uncertainty penalties can downweight plans that pass through states the model does not understand.

This matters when deploying RL systems because the planner will search for loopholes in its internal world. If a supply-chain model underestimates delays, the policy may route inventory through fragile paths. If a simulator omits actuator latency, a robot policy may look elegant in simulation and unstable on hardware.