Direct Preference Optimization

Is there a faster way than training a reward model and then running PPO? Often, yes: skip the separate reward model in the training loss and optimize the policy directly from preference pairs.

Direct Preference Optimization, or DPO, uses a single supervised-style objective over a prompt $x$ , a preferred answer $y^+$ , a rejected answer $y^-$ , a reference model $\pi_{ref}$ , and the current model $\pi_\phi$ .

For each pair, define the model's log-odds preference for the chosen answer:

$\Delta\log\pi_\phi=\log\pi_\phi(y^+\mid x)-\log\pi_\phi(y^-\mid x)$

and the reference model's log-odds:

$\Delta\log\pi_{ref}=\log\pi_{ref}(y^+\mid x)-\log\pi_{ref}(y^-\mid x)$

The question is: how much more does the current model prefer the chosen answer compared with the reference model?

Direct Preference Optimization

Compare chosen-versus-rejected log odds

DPO treats each preference pair like a supervised example, then asks whether the current model prefers the chosen answer more than the reference model did.

Preference pair

x: Explain why KL control matters during RLHF.

Preferred y+

Reward pulls the policy toward helpful outputs, while KL keeps it near the trusted reference.

Rejected y-

Just maximize the reward score as much as possible.

What the subtraction means

Δlog π_φ measures how much the current model prefers y+ over y-.

Δlog π_ref measures how much the reference model already preferred y+ over y-.

DPO trains on the difference between those two margins: did the current model improve beyond the reference?

Current model πφ

Move these to see how the trained model scores the same chosen/rejected pair.

log πφ(y+ | x)-0.8

log πφ(y- | x)-2.4

Chosen-answer share among this pair83%

Reference model πref

This is usually the SFT model. DPO uses it as the comparison baseline.

log πref(y+ | x)-1.4

log πref(y- | x)-2.1

Chosen-answer share among this pair67%

Current margin

Δlog π_φ = log π_φ(y+ | x) - log π_φ(y- | x) = 1.6

Reference margin

Δlog π_ref = log π_ref(y+ | x) - log π_ref(y- | x) = 0.7

DPO signal

0.9

Beta1.0

DPO objective signal: log σ(β · (Δlog πφ - Δlog πref))

The current model has learned a stronger preference for the chosen answer than the reference had.

With β = 1.0, the smooth win score is 0.71.

The reference model is the anchor. DPO rewards relative improvement over that anchor, not just raw confidence in the chosen answer.

The DPO objective maximizes:

$\mathbb{E}\left[\log\sigma\left(\beta(\Delta\log\pi_\phi-\Delta\log\pi_{ref})\right)\right]$

The sigmoid $\sigma(\cdot)$ acts like a smooth win score, and the log turns it into a stable loss. The temperature $\beta$ controls how sharply the objective separates winners from losers. The reference model is a built-in regularizer: it keeps style and safety anchors in the objective without writing an explicit KL term.

ℹ

DPO Intuition

If the policy already prefers $y^+$ over $y^-$ more than the reference does, then $\Delta\log\pi_\phi>\Delta\log\pi_{ref}$ and the term inside $\sigma(\cdot)$ is positive. If it prefers the chosen answer less than the reference does, the term is negative and DPO pushes the policy toward the preferred answer.

DPO friends

IPO

Inverse preference optimization with an alternative derivation and regularization view

ORPO

Preference optimization that adds constraints to preserve instruction-following

RRHF

Rank-based loss using listwise or ranking signals

KTO

Targets calibrated log-odds and preference margins

✦

Tips for DPO

Choose a strong reference model, often the SFT policy
Calibrate $\beta$ on held-out win-rate
Mix near-ties and adversarial pairs
Randomize order during data collection
Evaluate with win-rate, safety slices, and qualitative critique review

Checkpoint

What does DPO compare in its objective?

←PreviousPPO OptimizationReinforcement Learning with Human Feedback Next→Hybrid WorkflowsReinforcement Learning with Human Feedback