RL Fine-Tuning Overview

Reinforcement Learning with Human Feedback, or RLHF, fine-tunes a language model by optimizing a policy $\pi_\phi$ with a learned preference reward $\hat r_\theta$ , while keeping the policy close to a trusted reference model.

The high-level pipeline is: supervised fine-tuning creates a useful reference policy, preference data trains a reward model, and RL fine-tuning optimizes the current policy against reward minus drift.

RLHF pipeline

From demonstrations to constrained RL

RLHF fine-tuning turns human examples and preferences into a policy update that is useful but still anchored to a reference model.

Supervised Fine-Tuning

Learn a reference policy from high-quality human-written prompt-response pairs.

\pi_0(y\mid x)\leftarrow \text{human demonstrations }(x,y^*)

The three mathematical anchors are Bradley-Terry reward learning, the KL-regularized policy objective, and PPO-style clipped updates.

Supervised fine-tuning

SFT learns an initial policy $\pi_0(y\mid x)$ from human-written pairs $(x,y^*)$ . These demonstrations teach instruction following, answer format, refusal style, uncertainty language, and other behaviors worth preserving.

$\mathcal{L}_{SFT}(\theta)=-\mathbb{E}_{(x,y^*)\sim D}\left[\log \pi_\theta(y^*\mid x)\right]$

After SFT, $\pi_0$ is useful as both the starting policy and, often, the reference policy $\pi_{ref}$ used during RLHF or DPO.

Supervised fine-tuning

Choose the policy RL starts from

The same reward model update behaves differently depending on the starting policy.

Sample response

Thursday at 2 PM works. Here is the Zoom link. Thanks for coordinating.

Valid instructions86%

Reference drift16%

RL instability risk26%

The policy starts near useful behavior, so preference optimization can focus on subtle quality improvements.

SFT gives RLHF a reasonable reference policy: useful enough to optimize, stable enough to constrain against.

RL fine-tuning

RL fine-tuning then optimizes $\pi_\phi$ using the learned reward $\hat r_\theta(x,y)$ while staying close to the reference. The optimizer is not simply chasing reward; it is maximizing predicted human preference under a drift constraint.

That distinction matters because reward models are proxies. They help translate human judgments into gradients, but they can also reward shortcuts if optimization pressure is too strong.

ℹ

Three Anchors

Bradley-Terry: learn a scalar reward model from chosen/rejected pairs
KL objective: maximize reward minus deviation from a reference
PPO: optimize the objective with clipped, stochastic policy-gradient updates

Checkpoint

What is the main role of SFT in an RLHF pipeline?

←PreviousEvaluation and Safety AuditsHuman-in-the-Loop RL Next→Preference Data for RLHFReinforcement Learning with Human Feedback