Distributional Mismatch

Distributional mismatch is the main problem in offline RL. A learned policy is optimized by looking for actions that maximize $Q(s,a)$, but many actions the policy considers were never taken in the dataset. To evaluate those actions, the value function must extrapolate outside the data distribution, and extrapolation is unreliable.

This is the core difference from ordinary supervised learning. A supervised model usually predicts $y$ from inputs $x$ that resemble the training set. Offline RL must reason about counterfactuals: What would have happened if a different action had been taken? What reward would have occurred? How would future states have evolved? Those questions require extrapolation, not just interpolation.

Bootstrapped value learning makes this especially delicate. A value update often uses a target like

$$Q(s,a)\leftarrow r+\gamma\mathbb{E}_{s'}[V(s')]$$

If $Q(s,a)$ is estimated for an unsupported action, the immediate reward, next-state distribution, and future value may all be extrapolated. Neural value functions can assign arbitrarily high Q-values to unfamiliar state-action pairs because the data never constrained those predictions. The policy then exploits these errors and selects actions the dataset provides no evidence for.

A soft policy improvement step makes the amplification visible:

$$\pi(a\mid s)\propto \exp(Q(s,a))$$

A slightly inflated $Q$ can become a much larger action probability. Even tiny overestimates get magnified, pushing the policy deeper into out-of-distribution regions where $Q$ becomes even less grounded.