Sparse, Dense, Shaped, and Proxy Rewards
Reward signals differ in how often they provide feedback and how directly they measure the real objective. A sparse reward appears only at rare success points, such as reaching a goal. A dense reward gives frequent feedback, such as progress toward the goal at each step.
Reward shaping adds intermediate hints, such as rewarding a first-time door opening before the final goal. A proxy reward uses a measurable stand-in for something harder to measure, such as clicks for user satisfaction or simulated congestion for chip performance.
Reward type
What signal does the agent actually receive?
Use the same maze goal. Change only the reward type and inspect the tradeoff.
Reward signal
Only give +1 when the agent reaches the goal.
What it helps
Cleanly measures final success.
What can go wrong
The agent may wander for a long time before seeing any learning signal.
Sparse rewards are clean but hard to discover. If the agent almost never reaches the goal by chance, it may receive no useful learning signal. Dense and shaped rewards can make learning easier by giving feedback along the way.
The tradeoff is incentive risk. Every extra reward term is something the agent can optimize. If a door-opening bonus can be triggered repeatedly, the agent may farm the door bonus and stop exploring. If a proxy is easier to improve than the real goal, the agent may improve the proxy while the real system gets worse.
Proxy Is Not Reality
Proxy metrics are useful because real goals are often hard to measure. They are dangerous because the proxy can come apart from the real objective under optimization pressure.
When can a denser or shaped reward make learning easier but behavior worse?