Introduction to Modern Reinforcement Learning

IRL needs additional principles to choose a good reward from many possible explanations. Maximum Entropy IRL, introduced by Ziebart in 2008, adds one such principle: among policies that match expert feature expectations, choose the one with maximum entropy.

The idea is to explain the demonstrations without becoming overconfident. Humans are variable. Experts are not deterministic machines. MaxEnt IRL assigns higher probability to higher-reward trajectories while still allowing multiple plausible ways to act.

Equation walkthrough

Maximum Entropy IRL

Click each equation to see how MaxEnt IRL handles reward ambiguity without forcing one deterministic explanation.

Trajectory probability

higher-reward trajectories are more likely

Maximum Entropy IRL assigns more probability to trajectories with higher total reward while still allowing non-optimal or varied demonstrations.

Think: expert behavior minus your model's behavior, measured in feature space.

MaxEnt IRL models the probability of a trajectory as proportional to the exponential of its total reward:

$P(\tau)\propto\exp\left(\sum_t r(s_t,a_t)\right)$

The log-likelihood of expert demonstrations is:

$\mathcal{L}=\sum_{\tau_E}\log P(\tau_E)$

Training maximizes this likelihood with respect to reward parameters. The compact objective is:

$\mathcal{L}=\mathbb{E}_{\tau\sim\pi_E}[r(\tau)]-\log Z$

Here $Z$ is the partition function, a normalization constant that ensures trajectory probabilities sum to 1.

The gradient has a useful interpretation.

Equation walkthrough

MaxEnt IRL gradient

Click each term to see why the gradient becomes expert features minus model features.

Gradient of the likelihood

which direction should the reward parameters move?

The gradient tells us how to change the reward model so expert demonstrations become more likely.

If the model under-produces an expert feature, the reward weight for that feature should increase.

IRL becomes feature expectation matching. The update is expert behavior minus the model's behavior in feature space. If expert trajectories contain more "safe distance" than the model's trajectories, increase reward weight on safety. If the model overuses speed relative to the expert, reduce that explanation.

ℹ

MaxEnt Leads to Stochasticity

Entropy encourages exploration where demonstrations are uncertain, captures variability in human demonstrations, and avoids brittle policies that pretend every expert action was the only acceptable action.

This works well in small, tabular environments, but high-dimensional tasks expose three problems. First, classic IRL often requires hand-crafted features $f(s,a)$ , which collapse in image-based robotics, locomotion, and messy real-world settings. Second, rewards learned from narrow training trajectories may drift in new states, such as different lighting, posture, or dynamics. Third, matching feature expectations can require running RL inside every gradient step, which becomes expensive, noisy, and unstable in deep RL.

These limitations motivate adversarial approaches such as AIRL, which replace hand-crafted feature matching with a discriminator trained using deep networks.