States, Actions, Rewards, and Policies

Let's consider a small maze, in which a mouse must navigate to receive a cheese reward.

State $s$ is what the agent observes. In the mouse maze, the state might be the mouse's current position and nearby walls. Action $a$ is a choice available from that state: move left, right, up, or down. Reward $r$ is immediate scalar feedback: cheese might be $+10$ , a dead end might be $-1$ , and an ordinary move might be $0$ . Policy $\pi$ maps states to actions.

State, action, reward, policy

Run policies through a tiny mouse maze

Use the same maze to test different policies under different rewards. The policy is the behavior rule; the reward is the score the agent is optimizing.

Candidate policy

Reward to optimize

🐭

🧀

Total reward

10.0

Steps

Reached goal

yes

Action trace

right -> right -> right -> right -> down -> down -> down -> down

Candidate policy and reward are independent. Change the policy to compare behaviors; change the reward to rescore the same behavior.

A policy that scores well under one reward can be useless under the intended goal. This is the first safety lesson in the course.

With those definitions, an RL problem becomes a design problem. What should the agent observe? Which actions should it be allowed to take? What reward will actually be optimized? What policy class can represent competent behavior?

Reward examples can look silly and still reveal a serious problem. A robot rewarded for cleaning may create new messes so it can clean more. A game-playing boat may circle through point boosters and leave the race unfinished. A mouse rewarded only for moving fast may miss the cheese.

Reward hacking examples. — Reward Hacking (Source: EvilAICartoons.com)

⚠

Reward Encodes Incentives

The intended goal lives in the designer's head. The specified reward lives in the environment. The agent optimizes the specified reward.

Video: states and actions in reinforcement learning.

Video: states, actions, rewards, and policies.

Checkpoint

What is a policy in the mouse maze example?

←PreviousWhat Is an Agent?From AlphaZero to RLHF Next→AlphaZero as a Motivating ExampleFrom AlphaZero to RLHF