Unit 4

Advanced RL Methods

Dive into advanced RL techniques: offline learning, model-based agents, world models, hierarchy, inverse RL, and reward inference.

Chapter 1

Offline RL

Learn policies from fixed datasets without new environment interaction, and study why out-of-distribution actions become central.

1. Learning from Fixed Datasets→
2. Distributional Mismatch→
3. Conservative Objectives→
4. Decision Transformers→

Chapter 2

Model-Based RL and World Models

Study agents that learn models of the environment, plan through learned dynamics, and use world models for efficient policy learning.

1. Learning Transition Models→
2. Planning with Learned Dynamics→
3. Dreamer→
4. DreamerV3→
5. MuZero→
6. Sim-to-Real→

Chapter 3

Hierarchical RL

Use temporal abstraction, options, skills, and subgoals to reason about long-horizon tasks.

1. Temporal Abstraction→
2. Options and Skills→
3. Subgoals→
4. HIRO→
5. Open Research Challenges→

Chapter 4

Inverse RL and Reward Inference

Infer objectives from observed behavior, then examine imitation, ambiguity, maximum entropy IRL, adversarial reward learning, and preference-based reward inference.

1. Inferring Rewards from Behavior→
2. Apprenticeship Learning→
3. Reward Ambiguity→
4. Maximum Entropy IRL→
5. AIRL→
6. Preference-Based Reward Inference→