Reward Ambiguity

Reward ambiguity is the reason inverse RL isn't a perfect solution. Many reward functions can explain the same behavior. If an expert takes a longer route, they might be avoiding danger, saving battery, following a rule, respecting a social norm, or simply acting out of habit. Observing the route alone does not reveal which explanation is correct.

Reward ambiguity

Add evidence until one reward story breaks

The same demonstration can support several objectives. Toggle extra observations and watch which reward hypotheses remain plausible.

Expert detours around wet floorExpert waits for a pedestrianExpert takes shortcut when aloneExpert signals before crossing

Safety reward hypothesis64%

Speed reward hypothesis59%

Politeness reward hypothesis42%

Evidence items

Ambiguity

78%

Several reward stories still explain the behavior. You need interventions, counterexamples, or new contexts before deploying the inferred reward.

Reward ambiguity is not a footnote. Inferred rewards should be treated as hypotheses that need stress tests.

Ambiguity changes as new observations arrive. A detour around a wet floor supports a safety hypothesis. Waiting for a pedestrian supports safety and politeness. Taking a shortcut when alone supports speed but does not necessarily reject safety. Each observation shifts the likelihood of several reward stories.

Formally, IRL often asks for a reward under which expert behavior is near optimal:

$R^*\in\arg\max_R P(D_{\text{expert}}\mid R)$

The problem is that many rewards can give high likelihood to the same demonstration dataset $D_{\text{expert}}$ . Some rewards differ only by shaping terms. Others make different predictions in new situations.

The inverse problem is underdetermined. If the expert always walks straight, the reward could mean "walking straight is good," "turning is bad," "standing still is worse," or even an accidental preference for floor tiles in that direction. Infinitely many rewards can make the same behavior appear optimal.

Equation walkthrough

Why reward recovery is underdetermined

Click each term to see why many rewards can make expert behavior look optimal.

for

Expert expected reward

reward assigned to demonstrations

The recovered reward should score expert trajectories highly. But many different reward functions can do that.

IRL needs extra principles because expert behavior alone usually does not identify one unique reward.

Ambiguity should change how teams deploy reward inference. Do not treat an inferred reward as ground truth. Test it on counterfactual scenarios: What would the policy do if the shortest route is unsafe? What if following the human demonstration conflicts with a written rule? What if the expert acted under time pressure? What if the training data underrepresents a user group?

ℹ

How to Reduce Ambiguity

Collect demonstrations across varied contexts, add preference comparisons, ask experts to explain edge cases, include counterexamples, evaluate policies under distribution shift, and preserve uncertainty over reward hypotheses instead of collapsing too early to one scalar reward.

Checkpoint

Why is reward ambiguity especially important for alignment?

←PreviousApprenticeship LearningInverse RL and Reward Inference Next→Maximum Entropy IRLInverse RL and Reward Inference