Inferring Rewards from Behavior

Inverse reinforcement learning reverses the usual RL problem. Standard RL starts with a reward function and asks the agent to find behavior that maximizes it. Inverse RL starts with behavior and asks what reward function would make that behavior sensible.

We rarely articulate our reward functions. Why did a person brake early? Why do doctors choose treatment A over treatment B? Why does an expert pilot fly that trajectory? Behavior reveals preferences, priorities, and constraints. IRL asks: can we infer these?

In standard RL, the world gives us the reward. Given environment dynamics, a reward function $r(s,a)$, and discount factor $\gamma$, the agent computes or approximates an optimal policy:

$$\pi^*=\arg\max_\pi\mathbb{E}_\pi\left[\sum_t\gamma^t r(s_t,a_t)\right]$$

This is the forward direction: reward $\rightarrow$ behavior.

In IRL, the reward is unknown. Instead, we observe expert trajectories $\tau_E=\{(s_t,a_t)\}$ and search for a reward under which the expert looks approximately optimal:

$$\pi_E\approx\arg\max_\pi\mathbb{E}_\pi\left[\sum_t\gamma^t r(s_t,a_t)\right]$$

The expert's actions should look optimal under the reward we recover. Conceptually, IRL tries to match intentions, or the principles that justify observed actions, rather than merely matching actions.

The path-tracing activity shows the core inference problem. A short path to the goal might suggest a speed reward. A detour around the wet floor might suggest a safety reward. A stop near the charger might suggest battery management. The observed behavior is evidence, not proof. The same path can be compatible with several objectives.

A common maximum-entropy framing assigns higher probability to trajectories with higher reward while still allowing imperfect behavior.

The trajectory $\tau$ is the observed sequence of states and actions. The inferred reward $R$ should make expert demonstrations more likely than alternatives, but the model should not assume the expert is a perfect optimizer in every state.

Why not just use behavior cloning? Behavioral cloning asks, "What action would the expert take here?" It can be simple and stable when the deployment distribution closely matches the demonstrations, but it is brittle outside the demo distribution, can replicate mistakes, has no understanding of why the behavior is good, and learns no reward model that can be re-optimized.

IRL asks, "What objective seems to explain the expert's choices?" If an expert avoids a cliff, behavior cloning learns something like "the expert pressed left here." IRL tries to infer "cliffs are costly." That inferred reward can then be optimized in new states, used to compare alternative policies, adapted to new dynamics, or inspected for alignment problems.

If we recover a useful reward, we can do more than replay the demonstrations. We can improve on an imperfect expert, transfer to new dynamics, plan under different constraints, examine what the inferred model values, and detect when the inferred reward is misaligned. This is one reason reward inference sits underneath many modern alignment strategies, including RLHF-style reward modeling.