Unit 3

Human Feedback and Alignment

Connect reinforcement learning to human preference data, reward modeling, and policy optimization used in aligned AI systems.

Chapter 1

Human-in-the-Loop RL

Study how humans and AI raters shape RL systems through preference data, constitutions, learned reward models, disagreement-aware training, and safety evaluation.

1. The Human Feedback Loop→
2. The RLHF Pipeline→
3. Bradley-Terry Reward Model→
4. Disagreement is Signal→
5. RLAIF and Constitutional AI→
6. Evaluation and Safety Audits→

Chapter 2

Reinforcement Learning with Human Feedback

Walk through RLHF from supervised fine-tuning and Bradley-Terry reward learning to KL-controlled PPO, DPO, hybrids, failure modes, and practical troubleshooting.

1. RL Fine-Tuning Overview→
2. Preference Data for RLHF→
3. Reward Model and KL Objective→
4. PPO Optimization→
5. Direct Preference Optimization→
6. Hybrid Workflows→
7. Failure Modes and Mitigations→
8. Toolbox and Troubleshooting→