Double DQN

DQN variants are designed to solve specific problems. Double DQN targets overestimation. Dueling DQN targets inefficient value learning when many actions have similar consequences.

Overestimation happens because $\max_a Q(s,a)$ tends to pick actions whose estimates are accidentally too high. If all Q-values contain noise, the maximum is biased toward positive noise.

DQN variants

Match the variant to the problem

Choose the training problem. The relevant DQN variant appears with the idea it changes.

Use

Double DQN

a^* = \arg\max_a Q(s',a;\theta),\quad y = r + \gamma Q(s',a^*;\theta^-)

The online network selects the next action. The target network evaluates that selected action. This reduces the upward bias caused by taking a max over noisy estimates.

Double DQN changes target construction. Dueling DQN changes the network architecture.

Double DQN reduces overestimation by decoupling action selection from action evaluation. The online network chooses the best next action:

$a^*=\arg\max_a Q(s_{t+1},a;\theta)$

The target network evaluates that chosen action:

$y_t=r_{t+1}+\gamma Q(s_{t+1},a^*;\theta^-)$

This matters because the same noisy estimates are no longer used for both choosing and evaluating the maximum. The online network still picks the action, but the slower target network supplies the value used in the Bellman target.

Equation walkthrough

Double DQN separates choosing from evaluating

Click each line to see how Double DQN changes the target construction.

standard DQN

Double DQN selects

then evaluates

target

Online selection

choose the next action

The online network chooses which next action currently looks best.

Terms

The key move is decoupling: the online network selects the action, and the target network evaluates it.

ℹ

What Double DQN Changes

Double DQN changes target construction. It does not change the basic DQN idea of learning action values from replayed transitions.

Checkpoint

What problem is Double DQN mainly designed to reduce?

←PreviousTarget NetworksValue-Based Deep RL Next→Dueling DQNValue-Based Deep RL