Dreamer

World models push model-based RL toward a powerful idea: the agent does not have to plan in raw observation space. Instead, it can learn a compact latent state that captures what matters for control, then train behavior inside that learned latent world. Dreamer-style agents are the clearest example of this approach.

Imagine a robot learning from camera images. Predicting every future pixel is expensive and often unnecessary. The policy may not need the exact wall texture, the reflection on the floor, or the color of a background object. It needs to know where the gripper is, where the object is, whether contact has happened, whether the episode is likely to continue, and which actions move it toward reward. Dreamer learns that kind of internal world and uses imagined latent trajectories to update the actor and critic.

Dreamer world model visualization 1 — Dreamer learns a world model from experience. Using past images (o1–o3) and actions (a1–a2), it computes a sequence of compact model states (green circles) from which it reconstructs the images (ô1–ô3) and predicts the rewards (r̂1–r̂3). [Source]

Latent imagination

Build the world Dreamer trains inside

Choose what the latent state remembers. Dreamer-style agents learn a compact world model, then improve actor and critic on imagined trajectories.

Latent keeps

positionvelocitycontactsgoal distancecontinue signal

Latent drops

background cluttercamera noisedecorative texture

Latent consistency86%

Reward prediction82%

Actor learning signal88%

Dreamer loop

1. Encode real observations.
2. Train latent dynamics.
3. Imagine rollouts.
4. Update actor and critic.

This is the Dreamer idea: learn a compact world where imagined rollouts train useful behavior.

A world model is successful when its imagined futures preserve the control problem, not when it reconstructs every sensory detail.

Dreamer long-term prediction visualization — Predicting ahead using compact model states enables long-term predictions in complex environments. Shown here are two sequences that the agent has not encountered before. Given five input images, the model reconstructs them and predicts the future images up to time step 50. [Source]

Dreamer long-sighted behavior visualization — Dreamer learns long-sighted behaviors from predicted sequences of model states. It first learns the long-term value (v̂2–v̂3) of each state, and then predicts actions (â1–â2) that lead to high rewards and values by backpropagating them through the state sequence to the actor network. [Source]

The training loop is cyclic. First, the agent collects real experience. Second, it updates the world model so latent predictions match observed dynamics, rewards, and episode continuation. Third, it starts from encoded real states and imagines many possible future trajectories. Fourth, it improves the actor and critic using those imagined trajectories. Then the improved policy collects new real experience, and the cycle repeats.

Dreamer continuous control performance visualization — TOP: Dreamer learns to solve 20 challenging continuous control tasks with image inputs, 5 of which are displayed here. The visualizations show the same 64x64 images that the agent receives from the environment. BOTTOM: Dreamer learns successful behaviors on Atari games and DeepMind Lab levels, which feature discrete actions and visually more diverse scenes, including 3D environments with multiple objects. [Source]

Dreamer Atari and DeepMind Lab performance visualization — TOP: Dreamer learns to solve 20 challenging continuous control tasks with image inputs, 5 of which are displayed here. The visualizations show the same 64x64 images that the agent receives from the environment. BOTTOM: Dreamer learns successful behaviors on Atari games and DeepMind Lab levels, which feature discrete actions and visually more diverse scenes, including 3D environments with multiple objects. [Source]

World models usually have a three-part architecture: an encoder compresses observations into a latent state $z_t$ , a dynamics model predicts future latent states, and a controller chooses actions from the latent state. The critical piece is to predict in latent space rather than pixel space.

Pixel-space prediction	Latent-space prediction
High-dimensional and sensitive to tiny visual errors.	Compresses state and filters irrelevant noise.
May spend capacity on textures, backgrounds, and details irrelevant to control.	Can focus on causal structure: position, contact, velocity, reward, and continuation.
Long-horizon predictions can drift as pixel errors compound.	Rollouts can be cheaper and more stable when the latent state is well validated.

ℹ

Why Latent Space Helps

Latent models can reduce noise, make rollouts cheaper, and focus learning on causal task structure. They are especially useful when observations are high-dimensional but the control problem depends on a smaller set of hidden variables.

Checkpoint

Why can learning in latent space be more practical than predicting raw pixels directly?

←PreviousPlanning with Learned DynamicsModel-Based RL and World Models Next→DreamerV3Model-Based RL and World Models