Target Networks

DQN learns by bootstrapping: it updates one prediction using another prediction. The online network predicts $Q(s_t,a_t;\theta)$, and the target includes a future value estimate for $s_{t+1}$.

If the same network supplies both sides, the target changes every time the network learns. The agent is then chasing a target that moves because of its own last update.

DQN stabilizes this by keeping two networks. The online network, $Q(s,a;\theta)$, is updated by gradient descent. The target network, $Q(s,a;\theta^-)$, is copied from the online network only every $C$ steps.

• Online network: $Q(s,a;\theta)$
• Target network: $Q(s,a;\theta^-)$
• Copy step: periodically update $\theta^-\leftarrow\theta$

The target network appears in the Bellman target:

$$y_t = r_{t+1} + \gamma \max_{a'} Q(s_{t+1},a';\theta^-)$$

Only the prediction $Q(s_t,a_t;\theta)$ is directly trained on the current gradient step. The target network creates a slower reference point for the future-value estimate.