Evaluation and Safety Audits

Goodhart's Law says that when a measure becomes a target, it ceases to be a good measure. In RLHF, the reward model is the measure and policy optimization turns it into the target. That can create sycophancy, deception incentives, reward hacking, blandness, or excessive refusal under naive preference signals.

KL control

During RLHF, the reward model gives the optimizer a direction: make outputs that score higher. But if the policy chases that reward too aggressively, it can drift away from the supervised fine-tuned model that already behaved reasonably. KL control adds a penalty for moving too far from a reference policy, usually the SFT model.

The coefficient $\lambda$ controls the strength of that penalty. A larger $\lambda$ keeps the policy closer to the reference; a smaller $\lambda$ gives the reward model more room to reshape behavior.

Preference evaluation

A basic release metric is win-rate against one or more baselines:

$$WR=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\text{new model wins on item }i\}$$

The indicator $\mathbf{1}\{\cdot\}$ is 1 if the new model wins and 0 otherwise. Report per-domain win-rate and 95% confidence intervals, often with bootstrap resampling. Also report preference satisfaction on a held-out set that was not used to train or tune the reward model.

Aggregate win-rate is never the whole story. Report per-group scores $WR_g$ for groups such as novices, experts, or topic slices, and inspect gaps:

$$\text{Gap}=WR_{g_1}-WR_{g_2}$$

Gaps can expose value pluralism effects or reveal that an optimization helped one slice while harming another.

Rater reliability

Track rater reliability so you know whether labels are stable enough for the conclusion you want to draw. Krippendorff's $\alpha$ is one common reliability measure:

$$\alpha=1-\frac{D_o}{D_e}$$

Here $D_o$ is observed disagreement and $D_e$ is disagreement expected by chance. Low reliability does not always mean the data is useless; it may mean the prompt is ambiguous, the rubric is underspecified, or different groups legitimately disagree.

Safety evaluations

Safety evaluations should include jailbreak and abuse tests, refusal appropriateness, and hallucination audits. Jailbreak tests use curated red-team sets and generative attacks; useful metrics include success-at-jailbreak, which should go down, and time-to-failure, which should go up.

Refusal appropriateness has two sides. Safe prompts should be answered, and unsafe prompts should be refused. Track over-refusal as $\text{refuse\_safe\_rate}$ and under-refusal as $\text{comply\_unsafe\_rate}$. Hallucination audits use fact-checked question-answer sets, citation checks, and contradiction tests.

Robustness and generalization

Off-distribution stress tests use paraphrases, adversarial prompts, and new domains. Report in-distribution and out-of-distribution win-rates:

$$\Delta=WR_{IID}-WR_{OOD}$$

A large gap means the model's apparent improvement may not transfer. For long-horizon, multi-turn tasks, track error compounding with metrics such as fail@$k$ and self-correction rate. A model that wins single-turn preferences can still fail when mistakes accumulate over a conversation.