Convergence Intuition

Q-learning can converge to $Q^*$ in tabular settings, with conditions. The intuition is repeated correction toward Bellman optimality across all state-action pairs.

The slogan is: keep exploring forever, but learn a bit less each time. Ongoing exploration keeps every table entry connected to evidence. A shrinking learning rate prevents late random samples from completely rewriting values that have already been estimated well.

A standard tabular convergence statement assumes conditions such as all $(s,a)$ pairs being visited infinitely often, a learning-rate schedule with $\sum_t\alpha_t=\infty$ and $\sum_t\alpha_t^2<\infty$, and $\gamma<1$ for discounted continuing tasks.

These assumptions explain why convergence becomes harder to reason about in deep RL. A neural network shares parameters across many states and actions, so updating one estimate can move many others at the same time. The table setting is clean because each state-action entry is explicit.