Introduction to Modern Reinforcement Learning

The second major case study moves from games to language models. A base language model predicts the next word or token. That objective can produce fluent text while leaving helpfulness, honesty, and harmlessness as separate goals.

InstructGPT shows how RL ideas enter modern AI systems. The pipeline starts with a pretrained model, adds supervised fine-tuning on human-written ideal responses, trains a reward model from human comparisons, and then optimizes the policy against that learned reward.

InstructGPT pipeline

From GPT-3 to preference-optimized behavior

Step through the RLHF pipeline used to turn a pretrained GPT-3 model into an instruction-following policy.

base model

demos

rankings

scores

policy

Selected stage

Start with GPT-3

Artifact: Pretrained GPT-3 language model
Training signal: Next-token prediction on a broad text corpus

The starting policy is GPT-3, a pretrained language model that has learned to predict the next token. This gives the model broad language ability before any instruction-following alignment step.

Output: A fluent base model that can continue text. Instruction following still needs additional training.

InstructGPT turns human demonstrations and rankings into training signals: first supervised imitation, then reward-model-guided policy optimization.

RLHF

Build a tiny reward model from choices

Choose the better response in each pair. Your choices become a small preference dataset, and the toy reward model uses them to score future responses.

A user asks for an exact answer to a question with missing information.

A user asks for a quick summary of a long policy document.

A user asks for instructions that could enable harm.

Optimization pressure65%

KL constraint strength38%

Learned reward weights

Helpful33%

Honest33%

Harmless33%

Careful answer

43.3

helpful 58, honest 92, harmless 78, drift 16

Refusal-heavy answer

37.5

helpful 24, honest 72, harmless 98, drift 12

Confident shortcut

29.0

helpful 98, honest 38, harmless 40, drift 24

Current policy update favors Careful answer. Change your labels or the KL constraint and notice how the winning behavior changes.

The key move in RLHF is turning human comparisons into a reward signal. The risk is that the learned signal is only a proxy for the behavior people actually want.

The reward model learns from comparisons. If response A receives score $r_A$ and response B receives score $r_B$ , a simple preference model can be written as:

$P(A \succ B)=\sigma(r_A-r_B)$

The sigmoid converts the score difference into a probability that A is preferred. Policy optimization then makes high-scoring responses more likely while usually constraining the model to stay close to the supervised fine-tuned reference policy.

Preference probability

Convert score difference into preference probability

A reward model can assign each response a scalar score. The sigmoid turns the difference between two scores into a probability.

Response A score1.2

Response B score0.1

Work the calculation

d = r_A - r_B = 1.2 - 0.1 = 1.1

P(A \succ B) = \sigma(d) = \frac{1}{1 + e^{-d}}

= \frac{1}{1 + e^{-1.1}}

= \frac{1}{1 + 0.333} = 0.750 = 75.0\%

Sigmoid curve reader

d = 1.1, P(A) = 75.0%

Score difference

1.1

A preferred

75.0%

B preferred

25.0%

A unlikely

d = -2 gives 12%

50/50

d = 0 gives 50%

A likely

d = +2 gives 88%

Only the difference matters. Raising both scores by the same amount leaves the preference probability unchanged.

⚠

Human Feedback Is Still a Proxy

RLHF optimizes a learned model of preferences collected from particular prompts, annotators, instructions, and candidate responses. The learned reward is a proxy for human values.

RLHF as Modern RL

From GPT-3 to preference-optimized behavior

Start with GPT-3

Build a tiny reward model from choices

Convert score difference into preference probability