Reward Hacking
Reward hacking happens when an agent finds a loophole in the reward function that maximizes return while the intended objective fails. It is often optimization succeeding against an incomplete specification.
Reward hacking is easiest to see when a metric can be increased without the real goal improving: toggling the same door for repeated bonus, maximizing five-minute likes with low-quality content, or improving traffic throughput in one direction while creating unfair waits elsewhere.
Reward hacking
Find the loophole
Choose a case, read the specified reward, then reveal the behavior that could score well while missing the intended objective.
Specified reward
+0.5 every time the agent toggles a door.
Intended objective
Explore new rooms and eventually reach the goal.
Loophole
Reveal after you predict the hack.
The debugging move is to separate four things: intended objective, specified reward, learned behavior, and independent success metrics. If reward improves while success metrics stay flat or get worse, the reward curve is deceiving.
Mitigations can include state-aware reward shaping, one-time bonuses for milestones, constraints, human feedback, multi-objective signals, and evaluation metrics kept separate from direct training rewards.
A Pretty Reward Curve Can Lie
If success is more important than reward, monitor task-specific KPIs such as goal completion, safety, fairness, retention, or real-world transfer. A reward curve going up and to the right still needs task evidence beside it.
Which observation is strongest evidence of reward hacking?