DreamerV3

DreamerV3 keeps the core Dreamer idea, but makes the world model more explicit and scalable. It separates recurrent memory, stochastic discrete representations, prediction heads, and actor-critic learning so the agent can train across visually complex domains with one coherent latent model.

For each timestep, an encoder maps sensory input to a representation $z_t$ , a recurrent sequence model maintains state $h_t$ from past actions, and the concatenation of $h_t$ and $z_t$ forms the model state used to predict reward, continuation, and reconstructed inputs when needed.

Equation walkthrough

DreamerV3 model components

Click each component to see how DreamerV3 builds and trains a latent world model.

Sequence model

Encoder

Dynamics predictor

Reward predictor

Continue predictor

Decoder

Sequence model

memory from past latent states and actions

The recurrent sequence model updates deterministic state h_t from the previous recurrent state, previous latent state, and previous action.

Pieces

The model state combines recurrent memory h_t with stochastic latent state z_t, then predicts dynamics, reward, continuation, and reconstruction targets.

DreamerV3 world model architecture — The world model encodes sensory inputs *x_t* using the encoder (enc) into discrete representations *z_t* that are predicted by a sequence model with recurrent state *h_t* given actions *a_t*. The inputs are reconstructed as *x̂_t* using the decoder (dec) to shape the representations. The actor and critic predict actions *a_t* and values *v_t* and learn from trajectories of abstract representations *z_t* and rewards *r_t* predicted by the world model. [Source]

Equation walkthrough

DreamerV3 world-model loss

Click each term to see how DreamerV3 balances prediction, dynamics, and representation learning.

≜[(++)]

Model objective

total DreamerV3 world-model loss

DreamerV3 trains the world model by minimizing a weighted objective over time, with terms for prediction, dynamics, and representation learning.

The expectation averages over latent states inferred by q_phi, while the sum trains the model across a sequence.

Earlier reconstruction-heavy world models had to reproduce visual details such as noise, backgrounds, and textures. Those details can make long-horizon imagination drift, and not all visual information is useful for control.

DreamerV3-style systems put more pressure on latent consistency, reward prediction, value learning, and continuation prediction. For each timestep, an encoder maps sensory input to a representation $z_t$ , a recurrent sequence model maintains state $h_t$ from past actions, and the concatenation of $h_t$ and $z_t$ forms the model state used to predict reward $r_t$ , continuation $c_t\in\{0,1\}$ , and reconstructed inputs when needed. Image encoders and decoders are usually convolutional networks; vector inputs often use MLPs; categorical latent variables can provide a compact discrete internal state.

DreamerV3 future prediction examples — A procedural maze and a quadrupedal robot are shown. Given 5 context images and the full action sequence of an unseen video, Dreamer predicts 45 frames into the future without access to intermediate images. From pixel observations, the world model learns an understanding of the underlying structure of each environment. [Source]

Checkpoint

Which set best describes the DreamerV3 world model components?

←PreviousDreamerModel-Based RL and World Models Next→MuZeroModel-Based RL and World Models