Introduction to Modern Reinforcement Learning

Human-in-the-loop reinforcement learning uses people as part of the learning system. Humans may provide demonstrations, compare model outputs, score answers with a rubric, critique mistakes, or audit behavior after policy optimization.

Human-in-the-loop RL

Follow the feedback loop

Human feedback enters as data, becomes a learned reward signal, shapes policy optimization, and then returns as evaluation.

Collect demonstrations, comparisons, ratings, critiques, or audit labels.

Watch for: Bias, anchoring, context leakage, unclear instructions.

The loop works only if evaluation can push back on the reward model. Humans are not just labelers at the beginning; they remain auditors after optimization.

Types of human preference data

Different feedback formats expose different signals. In the lab below, switch formats and complete the same kind of task a human annotator would see: choose between candidates, rank them, score axes, apply a rubric, critique an answer, explain a comparison, or make a pass/fail safety judgment.

Human preference data

Try the annotator task

Choose a data format, then complete the kind of judgment a human rater would actually make.

Prompt

Explain what a reward model does in RLHF to a classmate who knows supervised learning.

Pick the better answer for the prompt.

Recorded label

B preferred

Best for

Reward-model training, DPO, IPO.

Pro

Simple and consistent.

Con

Only ordinal.

The same prompt can become a different training signal depending on the interface: relative choice, ranked order, absolute score, structured rubric, critique, rationale, or threshold label.

Preferences as feedback

Many alignment problems are easier to judge than to specify. A person may not be able to write a reward function for "helpful, honest, and appropriately cautious," but they can often compare two answers to the same prompt and say which one better fits the situation.

Preference learning turns these judgments into training data. For a prompt $x$ and two candidates $A$ and $B$ , a rater gives a relative signal such as $A \succ B$ . This does not say that $A$ is perfect; it says that $A$ is better than $B$ for this prompt, audience, and rubric.

Pairwise preference

Judge the duel, then read the signal

Pairwise labels ask for a relative judgment. The reward model learns from the direction of the preference, not from a perfect numeric score.

Prompt

Explain LLMs to a city council evaluating AI policy.

Encoded training signal

B preferred

B better matches the audience and stakes.

The same pair can expose different tradeoffs: brevity, correctness, safety, audience fit, or usefulness.

ℹ

Preferences carry hidden criteria

When raters choose A over B, they may be weighting correctness, tone, safety, brevity, completeness, or audience fit. Good rubrics make those criteria visible so the model does not learn accidental patterns.

⚠

Common data collection failures

Watch for rater bias and anchoring, where the first answer sets the tone; context leakage, where prompt or formatting hints reveal the desired style; and poor annotator instructions, where raters silently invent different criteria.

✦

Collect higher-quality human feedback

Randomize candidate order, blind raters to model identity, provide concrete annotation instructions, track inter-rater reliability, and monitor label entropy. When uncertainty is high, collect more data or ask for short rationales instead of pretending the item is settled.

The Human Feedback Loop

Follow the feedback loop

Types of human preference data

Try the annotator task

Preferences as feedback

Judge the duel, then read the signal