Introduction to Modern Reinforcement Learning

Dueling DQN targets inefficient value learning when many actions have similar consequences. Instead of asking one stream to learn every full $Q(s,a)$ value separately, the network decomposes $Q$ into two parts.

$V(s)$ is the value stream: how good this state is. $A(s,a)$ is the advantage stream: how much better or worse action $a$ is compared to the average action in this state.

TLDR: the value stream judges how good the situation is, while the advantage stream judges how much each action matters.

Equation walkthrough

Dueling DQN splits situation value from action advantage

Click the pieces of the decomposition to see what each stream is responsible for.

=+(-)

parameters

Value stream

how good is this state?

The value stream outputs a single scalar: how good the situation is, regardless of which action you take.

Terms

TLDR: the value stream judges how good the situation is; the advantage stream judges how much each action matters.

Dueling DQN

Raw advantages

Step through the dueling calculation from raw advantages to normalized advantages to final Q-values.

Q (s, a; θ, α, β) = V (s; θ, β) + (A (s, a; θ, α) - (1/∣ A ∣) \sum_{a^{'}} A (s, a^{'}; θ, α))

The advantage head scores each action around the current state. These are not final Q-values yet; they only say which actions look better or worse relative to the others.

Up has the highest raw advantage (+2), right is next (+1), and left/down are worse (-1).

A={Left,Right,Up,Down}

A = 4

V(s) = 5

Raw advantages = [+2, +1, −1, −1]

Mean advantage = (2+1−1−1)/4 = 0.25

🧀

A=+2

A=-1

🐭

A=+1

A=-1

The full decomposition is

$Q(s,a;\theta,\alpha,\beta)=V(s;\theta,\beta)+\left(A(s,a;\theta,\alpha)-\frac{1}{|\mathcal A|}\sum_{a'}A(s,a';\theta,\alpha)\right)$

The advantage function outputs one value per action. In a four-action environment, it produces four advantage scores for the current state. Those scores answer: how much better or worse is this action compared to the average action here?

The parameters are split by role. $\theta$ is the shared representation learned before the streams separate. $\alpha$ belongs to the advantage head, which outputs one value per action. $\beta$ belongs to the value head, which outputs the single state value.

The average-advantage subtraction keeps the decomposition identifiable. $|\mathcal A|$ is the number of possible actions, $\sum_{a'}A(s,a';\theta,\alpha)$ sums advantages over all actions, and $\frac{1}{|\mathcal A|}$ turns that sum into the mean advantage. Without this normalization, the model could shift value into $V$ or $A$ in arbitrary ways while leaving the final $Q$ value unchanged.

ℹ

Why This Helps

If most actions are similar, it is useful to learn that the state itself is promising before the network has perfectly learned which action is slightly better. Dueling DQN lets the value stream carry that state-level signal.