Reward Model and KL Objective

The reward model converts pairwise judgments into a scalar scoring function $r_\theta(x,y)$. Given a prompt and two responses, it should assign the chosen response a higher score than the rejected one.

The common Bradley-Terry form turns a reward difference into a probability:

$$P(A\succ B\mid x)=\sigma\left(\beta(r_\theta(x,A)-r_\theta(x,B))\right)$$

This gives us a scalar reward model. The margin $r_\theta(x,A)-r_\theta(x,B)$ matters more than the absolute score, and $\beta$ controls how sharply the model separates winners from losers.

Once $\hat r_\theta$ exists, policy optimization tries to maximize predicted human reward. But in RLHF we also want the model to stay near behavior we already trust. That gives the KL-regularized objective:

$$J(\phi)=\mathbb{E}_{x,y\sim\pi_\phi}\left[\hat r_\theta(x,y)-\lambda D_{KL}\left(\pi_\phi(\cdot\mid x)\,\Vert\,\pi_{ref}(\cdot\mid x)\right)\right]$$

The reward term pulls toward human-preferred behavior. The KL term limits drift from the reference policy. The coefficient $\lambda$ controls the strength of the leash.

Adaptive $\lambda$

Most RLHF systems monitor a target KL distance $\kappa$. If observed KL rises above $\kappa$, raise $\lambda$ so the reference pulls harder. If observed KL stays far below $\kappa$, lower $\lambda$ so the policy can learn more from reward.

In short: $\hat r_\theta$ pulls toward better answers, while KL keeps the new policy from drifting too far too fast.