Bradley-Terry Reward Model

The goal of reward modeling is to learn a scalar reward $r_\theta(x,y)$ that explains human pairwise choices. The input is a prompt $x$ and a candidate answer $y$ ; the output is one number. During RL, the policy searches for answers that receive high reward.

For a prompt $x$ and two candidates $A$ and $B$ , humans pick one. If the human chooses $A$ , the reward model should assign $A$ a higher score than $B$ . What matters is the difference:

$\Delta r = r_\theta(x,A)-r_\theta(x,B)$

If $\Delta r > 0$ , then $A$ should be more likely to be chosen. The absolute scores are less important than their margin.

Reward model math

Tune the Bradley-Terry scoreboard

Move the reward scores and temperature. The preference probability depends on the difference, not the absolute score level.

Reward for A1.20

Reward for B0.10

Temperature beta1.40

P(A \succ B \mid x)=\sigma(1.4(1.2-0.1))=0.82

Delta r

1.10

P(A wins)

82%

loss if A chosen

0.19

Adding the same constant to both rewards changes neither the margin nor the predicted preference.

Bradley-Terry / logistic model

A common model converts the reward difference into a probability using the sigmoid function:

$P(A \succ B \mid x)=\sigma(\beta \cdot \Delta r)=\frac{1}{1+e^{-\beta\Delta r}}$

The parameter $\beta$ acts like an inverse temperature. Larger $\beta$ makes the same reward difference produce a more confident preference probability. Because reward scale and $\beta$ interact, practical systems fix the scale with regularization, calibration targets, or a reference distribution.

If $A$ was chosen, the one-example loss is:

$\ell=-\log \sigma(\beta\cdot\Delta r)$

ℹ

Gradient intuition: push and pull

Let $p=\sigma(\beta\cdot\Delta r)$ . For an example where A was chosen, gradient descent raises the reward for the winner and lowers the reward for the loser:

$\frac{\partial \ell}{\partial r_A}=-\beta(1-p)$

$\frac{\partial \ell}{\partial r_B}=\beta(1-p)$

If the model is wrong, then $p\approx0$ , so $1-p\approx1$ and the update is large. If the model is already confident, then $p\approx1$ , so $1-p\approx0$ and the update is tiny.

Only differences matter

Pairwise reward modeling is shift-invariant. Suppose we add a constant $c$ to every reward:

$r'(x,y)=r(x,y)+c$

Then the difference is unchanged:

$\Delta r'=(r(A)+c)-(r(B)+c)=r(A)-r(B)$

So the loss does not change. Pairwise comparisons teach relative reward, not an absolute human-value meter.

Reward model proxy

Watch the proxy pick a winner

Change the learned reward model's hidden bias. The policy will chase whichever candidate receives the highest scalar score.

Sycophantic answer

6.2

Reward score62%

Long answer

9.4

Reward score94%

Safe answer

5.8

Reward score58%

The policy can learn to pad answers because the proxy mistakes length for quality.

Reward models are useful precisely because they are optimizable. That is also why proxy errors become important under pressure.

⚠

Scale is not harmless

Even though pairwise labels are shift-invariant, scale still matters once the reward model is used for optimization. Overconfident scores can make the policy chase tiny proxy quirks very aggressively.

Checkpoint

In the Bradley-Terry reward model, why does adding the same constant c to r(A) and r(B) not change the pairwise loss?

←PreviousThe RLHF PipelineHuman-in-the-Loop RL Next→Disagreement is SignalHuman-in-the-Loop RL