The RLHF Pipeline
Reinforcement Learning with Human Feedback, or RLHF, trains a model to maximize a human-aligned learned reward. In language-model alignment, the pieces usually look like this:
- A pretrained model learns broad language patterns from next-token prediction.
- A supervised fine-tuned model learns to follow instructions from curated examples.
- A reward model learns to score candidate answers from human comparisons.
- A policy generates text.
- The reward model scores text, producing the scalar signal used during policy optimization.
The policy update is still reinforcement learning: actions are tokens, trajectories are generated responses, and the reward is a learned estimate of human preference rather than a hand-written environment reward.
Supervised fine-tuning
Choose the policy RL starts from
The same reward model update behaves differently depending on the starting policy.
Sample response
Thursday at 2 PM works. Here is the Zoom link. Thanks for coordinating.
Why SFT comes before RL
Supervised fine-tuning gives the policy a useful starting behavior. RLHF can then optimize subtle preferences instead of spending early updates discovering basic instruction-following, formatting, and refusal behavior from reward alone.
Comparison datasets
A comparison dataset usually contains a prompt, candidate outputs, a chosen answer, a rejected answer, the rubric version, rater metadata or group slice when appropriate, a timestamp, and sometimes a short rationale. The obvious goal is label accuracy, but the deeper goal is coverage: the dataset should teach the reward model the distinctions that matter during deployment.
That means collecting more than easy wins. Useful datasets include hard negatives, adversarial candidates, near ties, safety-sensitive requests, different user groups, gold checks for rater calibration, and examples where tradeoffs are real rather than decorative.
Preference dataset
Choose the training mix
A reward model can only learn distinctions that appear in the comparison data.
held-out accuracy
84%
calibration
82%
blind spot
Fewer cheap labels
Dataset curation
Fill the label cart
Toggle example types for a comparison dataset. A good cart is not just large; it covers hard cases and reveals disagreement.
Training data considerations
The reward model is only as good as the comparison distribution it sees. Dataset design should ensure prompt diversity, include hard negatives that are semantically similar to good answers but wrong in important ways, and include adversarial candidates that probe shortcut behavior.
Regularization also matters. Dropout, ensembling, uncertainty-aware losses, early stopping, and calibration checks can reduce overconfidence. These tools do not make the reward model "correct"; they make its limits more visible and less dangerous during policy optimization.
Easy labels can be too easy
If every pair has an obvious winner, held-out accuracy may look excellent while the model remains weak on the hard boundary cases that matter most: near ties, safety tradeoffs, adversarial prompts, and subgroup disagreements.
Why should a reward-model dataset include hard negatives and adversarial candidates?