Introduction to Modern Reinforcement Learning

Adversarial Inverse Reinforcement Learning, or AIRL, learns rewards through a competition. One part of the system tries to distinguish expert behavior from learner behavior. The other part tries to produce behavior that looks more expert-like. The reward signal emerges from this adversarial process.

The setup is useful when demonstrations exist but hand-designing a reward would be brittle. For example, an autonomous driving system might have expert trajectories through merges, stops, and lane changes. A simple reward like "maximize speed and avoid collisions" misses comfort, courtesy, anticipation, and legal nuance. AIRL uses the contrast between expert and learner trajectories to infer a reward-like signal that pushes the learner toward expert-style behavior.

ℹ

GAN Review

Generative Adversarial Networks use a game between two models: a generator that tries to produce convincing examples and a discriminator that tries to classify examples as real or fake. When the discriminator is correct, the generator is penalized. When the generator fools the discriminator, the discriminator is penalized.

AIRL uses the same idea for behavior. The policy plays the generator role by producing trajectories. The discriminator tries to tell expert transitions from policy transitions.

AIRL Training - Step Through the Adversarial Loop1 / 6

Setup

Collect Expert Demonstrations

Expert trajectories become the real examples. AIRL treats these transitions as evidence of the reward the expert appears to optimize.

Discriminator loss

not trained

No classifier boundary yet; only expert data has been observed.

Policy loss

not trained

The learner policy has not produced comparison rollouts yet.

Policy trajectory quality18%

Discriminator accuracy50%

Learned reward clarity12%

Key insight

Expert behavior is the positive class. AIRL still needs learner behavior as contrast before it can infer a useful reward signal.

AIRL replaces the classic feature-matching loop with adversarial classification. First collect expert demonstrations $\tau_E$ . Then collect rollouts from the current policy $\tau$ . Train the discriminator to classify expert versus policy transitions, extract a reward from the discriminator, update the policy using RL on the learned reward, and repeat until learner trajectories match the expert distribution.

A simplified discriminator objective assigns high scores to expert-like behavior and low scores to learner-like behavior.

Equation walkthrough

AIRL discriminator objective

Click each term to see how the discriminator learns from expert and learner transitions.

Train the discriminator

choose D to classify well

The discriminator D is the model being optimized here. Its job is to separate expert transitions from policy transitions.

The discriminator learns the contrast; the policy later uses that learned signal as reward.

In adversarial imitation methods, the policy can then receive a reward related to how expert-like its state-action pairs appear.

Equation walkthrough

AIRL's discriminator and reward structure

Click each term to see how AIRL separates task reward from dynamics-dependent shaping.

AIRL discriminator

probability transition is expert-like

The discriminator compares the learned reward-structure score exp(f) against the probability of the current policy taking action a in state s.

AIRL uses adversarial classification, but constrains the discriminator so the recovered reward can transfer better.

AIRL is especially interesting because it constrains the discriminator's score:

$f(s,a,s')=r(s)+\gamma V(s')-V(s)$

The reward term $r(s)$ is only allowed to depend on state. Any dependence on actions or transitions must be explained through the shaping term $\gamma V(s')-V(s)$ . This helps separate task objective from dynamics-related effects, so the recovered reward can transfer more cleanly to new environments.

Instead of requiring hand-crafted features, AIRL can use deep networks trained on raw states or state-action transitions. The discriminator automatically learns features that separate expert behavior from policy behavior, while the policy learns to fool the discriminator.

⚠

Discriminator Shortcuts

If the discriminator can distinguish expert from learner using irrelevant artifacts, the inferred reward can be misleading. Always ask what evidence the discriminator is using and whether that evidence will remain valid after deployment.