Failure Modes and Mitigations

Optimization amplifies whatever signal is easiest to exploit. If the reward model over-rewards a shortcut, PPO can make that shortcut much more common. KL control slows the movement, but evaluation and data refreshes still need to catch the failure.

Typical RLHF failure modes include reward hacking, blandness or mode collapse, sycophancy, and safety drift.

• Reward hacking: the policy exploits quirks in $\hat r_\theta$ that are not actually human-preferred
• Blandness / mode collapse: answers become safe but vacuous because the reward favors generic responses
• Sycophancy: the model agrees with the user because agreeable answers score well
• Safety drift: risky patterns reappear because the reward model did not catch a slice of behavior

KL control is the main pressure that keeps RLHF from turning every reward-model preference into a huge policy shift. The reward term says, "move toward outputs humans seem to prefer." The KL term says, "but stay close to the trusted reference model unless the reward gain is worth the distance."

In practice, this becomes a tuning question: how much movement should we allow before we worry the policy is learning reward-model quirks instead of better behavior?

Mitigation strategies

• Start with a tight KL leash and relax gradually
• Early stop if reward-model score rises while human win-rate is flat or falling
• Down-weight uncertain reward scores using ensembles or dropout variance
• Use diverse rollouts: multiple temperatures, adversarial prompts, and counter-prompts
• Periodically refresh with SFT examples to retain instruction-following style
• Evaluate by topic, user group, safety slice, and failure gallery