Introduction to Modern Reinforcement Learning

In many real settings, we do not have clean expert demonstrations. But we can often ask humans for comparisons: "A is better than B," "this trajectory looks safer," or "this answer is more helpful." That turns reward inference into preference learning.

Humans may struggle to specify rewards, but they can often compare two clips and say which feels safer, more competent, more moral, or more aligned with the task. This is the core move behind preference-based RL and many RLHF reward-modeling pipelines.

Equation walkthrough

Learning rewards from comparisons

Click each term to connect preference-based reward inference to the Bradley-Terry model.

Preference probability

Bradley-Terry choice model

The model predicts how likely a human is to prefer trajectory tau_1 over tau_2 based on their learned reward scores.

Humans may struggle to specify rewards, but they can often compare two clips and say which looks safer or better.

Given two trajectories $\tau_1$ and $\tau_2$ , a human selects the preferred one. A common Bradley-Terry choice model assigns preference probability according to the learned return of each trajectory:

$P(\tau_1\succ\tau_2)=\frac{\exp(R(\tau_1))}{\exp(R(\tau_1))+\exp(R(\tau_2))}$

The reward model $R(\tau)$ learns from comparisons. If trajectory $\tau_1$ repeatedly beats $\tau_2$ , the model learns to score $\tau_1$ higher.

ℹ

Connection to Unit 3

This is the same Bradley-Terry idea used in Unit 3's RLHF reward model. There, the model compares two responses to the same prompt and learns which one humans prefer. Here, the model compares two trajectories and learns which behavior humans prefer. In both cases, reward is learned from relative judgments rather than absolute scores.

The reward parameters are optimized by maximizing the likelihood of observed human choices:

$\mathcal{L}(\theta)=\sum \log P_\theta(\tau_i\succ\tau_j)$

After training, regular RL can optimize a policy under the learned reward. This separates two jobs: humans provide comparison data, and the RL algorithm searches for behavior that scores well under the learned reward model.

ℹ

Why This Matters for Alignment

Preference-based reward inference is a foundation for RLHF, preference optimization, modern reward modeling, and scalable oversight.

Reward inference connects the chapter's methods. IRL asks, "what is the expert optimizing?" Maximum Entropy IRL handles ambiguity with entropy. AIRL learns deep, more transferable rewards through adversarial training. Preference-based reward inference learns directly from human choices when demonstrations or explicit rewards are unavailable.

Preference-Based Reward Inference

Learning rewards from comparisons