Experience Replay

When an agent acts in an environment, nearby experiences are usually similar. If a mouse walks down a hallway, the next few transitions are all hallway transitions. If a game character is stuck near a wall, the newest frames may all describe the same local mistake.

Training a neural network only on the newest transition makes each update chase a narrow slice of experience. Experience replay fixes this by storing transitions in a buffer, often written as

$\mathcal D=\{(s_t,a_t,r_{t+1},s_{t+1})\}$

Experience replay

Training on only the newest steps is correlated

Choose how to build a four-transition training batch. The selected samples are highlighted in the timeline.

Replay buffer timeline

start

hall

door

goal

trap

t10

hall

t11

door

Correlation

high

These samples come from the same local stretch of behavior, so their errors tend to point in the same direction.

Replay buffers turn online experience into small supervised-learning batches: sample transitions, compute targets, update the network.

During learning, DQN samples a random minibatch from $\mathcal D$ , computes a Bellman target for each transition, and takes a gradient step on the average loss. This makes the update look more like ordinary supervised learning: a batch of examples, a prediction, a target, and an error.

Replay helps in three ways:

It breaks correlation between consecutive samples.
It increases data efficiency by reusing past experience.
It smooths learning updates so one recent stretch of behavior does not dominate the network.

⚠

Replay Does Not Make Data Independent

Replay only mixes the samples the agent has collected. If the policy never visits useful states, the buffer will not contain them. Exploration still matters.

Checkpoint

Why does DQN sample random minibatches from a replay buffer?

←PreviousDeep Q-NetworksValue-Based Deep RL Next→Target NetworksValue-Based Deep RL