Chapter 2

Reinforcement Learning with Human Feedback

Walk through RLHF from supervised fine-tuning and Bradley-Terry reward learning to KL-controlled PPO, DPO, hybrids, failure modes, and practical troubleshooting.

• 1. RL Fine-Tuning Overview→
• 2. Preference Data for RLHF→
• 3. Reward Model and KL Objective→
• 4. PPO Optimization→
• 5. Direct Preference Optimization→
• 6. Hybrid Workflows→
• 7. Failure Modes and Mitigations→
• 8. Toolbox and Troubleshooting→