Instability and Overestimation

Deep value-based RL is powerful because it combines three ingredients: function approximation, bootstrapping, and off-policy learning. The same combination can also become unstable.

We have trained our agent with DQN, but sometimes it forgets what it learned, oscillates, or even diverges. Why? Value estimation instability.

Function approximation means one gradient step can change many state-action estimates at once. Bootstrapping means targets depend on learned estimates. Off-policy learning means the agent may train on data collected by an older or more exploratory behavior policy.

DQN learns by bootstrapping: predicting from its own predictions. This creates instability because the target can drift as the Q-network drifts.

$$y_t=r_{t+1}+\gamma\max_{a'}Q(s_{t+1},a';\theta^-)$$

When DQN fails, the training curve may oscillate, collapse after apparent progress, or show Q-values growing far beyond plausible returns. The right diagnosis depends on the symptom.

A practical DQN loop therefore combines several safeguards: collect data with an exploration policy such as epsilon-greedy, store transitions in a replay buffer, sample random minibatches, compute targets with a target network, and periodically copy $\theta \rightarrow \theta^-$.

The temporal-difference error still drives the update. If the target is larger than the current prediction, the update increases $Q(s_t,a_t;\theta)$. If the target is smaller, it decreases the prediction. If the TD error is near zero, little changes.