Distribution Shift

Distribution shift means the deployment setting differs from the training setting. In supervised learning, this often means the inputs changed. In RL, the shift is more tangled because the policy affects which states become inputs at all.

A policy trained on a small grid might perform well on new random seeds from the same grid size, then fail when the map grows, the door moves, the action dynamics change, or obstacles appear. The agent has not merely seen different observations; it has entered a different trajectory distribution.

Distribution shift

Compare IID and OOD evaluation

The policy was trained on small door-key maps. Choose an evaluation setting and watch the success gap.

WR_IID

88%

WR_OOD

31%

Delta

57 pts

Sparse rewards and longer horizons expose whether the policy learned a strategy or memorized short paths.

In RL, the policy creates its own state distribution. A small environment change can send it into states it rarely saw while learning.

One compact way to report robustness is to compare in-distribution and out-of-distribution success:

$\Delta_{OOD}=WR_{IID}-WR_{OOD}$

Here $WR$ means win rate, or the percentage of evaluation episodes where the policy succeeds. $WR_{IID}$ is the win rate on in-distribution tests, and $WR_{OOD}$ is the win rate on out-of-distribution tests.

A small gap does not prove safety, but a large gap is a warning. It says the training return was partly tied to assumptions that do not survive the new setting.

ℹ

Policy-Induced Data

After deployment, a small mistake can push the agent into unfamiliar states. From there, later actions are chosen using estimates built from weak evidence, so errors can compound across the trajectory.

Checkpoint

Why is distribution shift especially sharp in RL?

←PreviousCuriosity-Driven LearningSafety, Generalization, and Exploration Next→Generalization FailuresSafety, Generalization, and Exploration