Unit 1

Foundations of Reinforcement Learning

Build the basic language of reinforcement learning: agents, environments, rewards, Markov decision processes, value functions, Bellman equations, tabular learning, and reward design.

Chapter 1

From AlphaZero to RLHF

Start with the central idea of RL: an agent learns by acting, observing consequences, and improving future decisions. AlphaZero, Gato, and RLHF provide concrete anchors for the rest of the course.

1. Why Reinforcement Learning Matters→
2. Agent-Environment Loop→
3. What Is an Agent?→
4. States, Actions, Rewards, and Policies→
5. AlphaZero as a Motivating Example→
6. Gato as RL-Generated Data→
7. RLHF as Modern RL→

Chapter 2

MDPs, Rewards, and Value Functions

Formalize sequential decision-making with Markov decision processes, then define rewards, returns, value functions, and Bellman equations.

1. Markov Decision Processes→
2. Rewards, Returns, and Discounting→
3. State-Value and Action-Value Functions→
4. Bellman Equations→

Chapter 3

Tabular RL and Q-Learning

Use small finite environments to understand learning from experience, balancing exploration and exploitation, and updating action values.

1. Planning vs. Learning→
2. Exploration vs. Exploitation→
3. Q-Learning Update Rule→
4. Convergence Intuition→

Chapter 4

Reward Design

Study how reward signals shape behavior, why proxies fail, and how reward design connects technical RL to safety and alignment.

1. Intended Goals vs. Specified Rewards→
2. Sparse, Dense, Shaped, and Proxy Rewards→
3. Reward Hacking→