Preference Data for RLHF

After SFT, RLHF needs examples of what "better" means. A comparison record usually contains a prompt $x$, a preferred answer $y^+$, a rejected answer $y^-$, a rubric version, rater metadata, and sometimes a short rationale.

The chosen answer should be better under the rubric, not merely longer, friendlier, or more confident. Good comparison data is what prevents the reward model or DPO objective from learning shallow shortcuts.

Different slices teach different lessons. Ordinary helpful prompts keep the model from becoming a refusal machine. Matched safe and unsafe prompts teach refusal appropriateness. Near ties teach calibration. Adversarial prompts teach robustness. Domain prompts teach that "good" depends on audience, expertise, and stakes.

A useful safety pair might compare a legal-access alternative against a procedural bypass answer. A useful honesty pair might compare "I do not know from the provided evidence" against a fluent hallucination. A useful brevity pair might make the concise answer the winner so the system does not confuse word count with value.

Data quality habits

• Mix easy wins, near ties, adversarial pairs, and domain-specific prompts
• Randomize A/B order and blind raters to model identity
• Include hard negatives that are semantically similar but wrong under the rubric
• Collect short rationales on contentious pairs
• Track rater reliability and keep held-out prompt families separate from training