Rewards, Returns, and Discounting

A reward is local feedback: what happened after one transition. A return is the accumulated consequence of a trajectory. The distinction matters because good behavior often requires looking beyond the next reward. A delivery robot might spend a few extra steps reaching an elevator, then complete the delivery much faster than if it chased the nearest hallway at every turn.

Rewards are often delayed. Working early on a project may feel unrewarding in the moment and later prevent an all-nighter. In a gridworld, each step might cost $-0.01$ while the goal gives $+1$ at the end. The agent needs a way to connect present actions to future consequences.

The discounted return from time $t$ is $$G_t=R_{t+1}+\gamma R_{t+2}+\gamma^2R_{t+3}+\cdots$$ The exponent grows with delay: one-step-later rewards are multiplied by $\gamma$, two-step-later rewards by $\gamma^2$, and so on.

$\gamma$ controls how much we discount the future. When $\gamma\approx 0$, the agent is myopic: it mostly cares about immediate reward. When $\gamma\approx 1$, the agent is farsighted: delayed rewards still have substantial weight.

Discounting changes comparisons. A short path to a small reward becomes more attractive when future rewards are heavily discounted. A delayed larger reward becomes more attractive when $\gamma$ is high enough for the future to still matter.