Introduction to Modern Reinforcement Learning

Tabular Q-learning works great when we can list every state-action pair. The agent stores one value for each pair, then updates that table as it learns.

But real-world problems do not look like tiny tables. Games, robotics, and language can have millions of states, high-dimensional observations, or continuous inputs. We cannot store a Q-table that big.

So DQN replaces the table with a function approximator, usually a neural network. The input is the state. The output is a predicted Q-value for each available action.

Instead of a lookup table, the agent learns parameters $\theta$ so that

$Q(s,a;\theta) \approx Q^*(s,a)$

DQN architecture

A Q-table becomes a Q-network

Choose an observation type. The output stays the same: one estimated Q-value for each available action.

State observation

84 x 84 pixels

Input shape: 7,056 numbers

Q-network

Q(s,a; theta)

Predicted action values

left

1.10

right

1.34

fire

2.41

stay

0.72

Greedy action

fire

The observation is too large to enumerate cleanly. A network maps the pixels to action values.

DQN is still value-based. The neural network replaces table lookup; it does not replace the Bellman target.

The network receives a state observation and outputs one predicted value for each discrete action. In an Atari game, the input might be pixels and the output might be values for left, right, fire, and stay. The greedy action is the action with the largest predicted Q-value.

DQN still learns from the Bellman optimality target. For a sampled transition $(s_t,a_t,r_{t+1},s_{t+1})$ , the target is usually written as

$y_t = r_{t+1} + \gamma \max_{a'} Q(s_{t+1},a';\theta^-)$

The prediction being trained is $Q(s_t,a_t;\theta)$ . The loss asks the online network to move that prediction toward the target:

$L(\theta)=\mathbb E\left[(y_t-Q(s_t,a_t;\theta))^2\right]$

Equation walkthrough

DQN trains a prediction toward a Bellman target

Click each term to see how the target and loss connect.

=+

=E[(-)²]

Bellman target

what the network should move toward

The target is immediate reward plus a bootstrapped estimate of the best future value.

Terms

The neural network predicts Q(s, a); the Bellman target tells that prediction where to move.

The update step adjusts the network weights in the direction that reduces prediction error. The temporal-difference error, or TD error, is the gap between the target value and the current Q estimate.

$\theta \leftarrow \theta + \alpha(y_t-Q(s_t,a_t;\theta))\nabla_\theta Q(s_t,a_t;\theta)$

If the TD error is zero, there is no update. If it is positive, the update increases $Q(s_t,a_t;\theta)$ . If it is negative, the update decreases that estimate.

Equation walkthrough

The TD error tells the Q-network which way to move

Click each part of the update rule to see how squared-error learning becomes a weight update.

←+(-)

TD error

target minus current estimate

If this is zero, there is no update. If it is positive, increase the current Q estimate. If it is negative, decrease it.

Terms

TD error = 0

no update

TD error > 0

increase Q

TD error < 0

decrease Q

This is gradient descent on squared TD error: if y_t - Q(s_t, a_t) is positive, Q moves up; if it is negative, Q moves down.

DQN still needs exploration, so it often chooses actions with an epsilon-greedy rule. With probability $\epsilon$ , the agent takes a random action. With probability $1-\epsilon$ , it takes the greedy action:

$a_t=\arg\max_a Q(s_t,a;\theta)$

A typical decay schedule is $\epsilon_t=\max(\epsilon_{min},\epsilon_0\times 0.995^t)$ , which starts exploratory and becomes greedier over time.

Exploration schedule

DQN still uses epsilon-greedy

Move through training time to see how random exploration decays while greedy Q-value action selection takes over.

Training stept = 250

random

29%

argmax Q

71%

epsilon

0.286

random action

29%

greedy action

71%

Typical schedule: ε_t = max(ε_min, ε₀ x 0.995^t).

Deep Q-learning algorithm

DQN turns interaction into supervised-style updates

Step through the training loop from acting to replay to target-network updates.

Initialize

Initialize Q-network weights theta, set target weights theta-minus equal to theta, and create an empty replay buffer.

theta = random weights

theta_minus = theta

D = empty replay buffer

The loop is simple in outline: collect experience, replay it in random minibatches, compute Bellman targets, and update the Q-network.

ℹ

What Changes From Tabular Q-Learning

The update target is familiar: immediate reward plus discounted future value. The representation changes. A DQN generalizes across similar observations instead of treating every exact state as a separate table row.

Deep Q-Networks

A Q-table becomes a Q-network

DQN trains a prediction toward a Bellman target

The TD error tells the Q-network which way to move

DQN still uses epsilon-greedy

DQN turns interaction into supervised-style updates