Policies as Trainable Objects

Value-based methods learn a map from situations to action values. DQN asks, "how good would each action be here?" Policy-gradient methods start from a different instinct: do not learn the whole value map first; directly tune the chooser.

A policy is written $\pi_\theta(a\mid s)$ , the probability that a policy with parameters $\theta$ takes action $a$ in state $s$ . In a discrete task, the policy might output probabilities for left, right, and wait. In a continuous-control task, it might output a steering angle or a distribution over motor commands.

PPO setup diagram showing a trainable policy producing actions, receiving rewards, and updating from sampled experience. — As opposed to value-based deep RL, which essentially learns a big map, PPO asks: can we get better at choosing by tuning the policy instead?

Policy gradients

What is the model learning?

Use the same state. Switch representations to see the difference between learning action values and learning the chooser directly.

State

A robot is one step from a charging dock. It can move left, move right, or wait.

\pi_\theta(a\mid s)

Output the chooser directly

left

18%

right

67%

wait

15%

The policy is the trainable object: changing the parameters changes the action probabilities themselves.

Policy-gradient methods optimize the parameters of the behavior policy itself: π_θ(a | s).

The objective is expected return under the current policy:

$J(\theta)=\mathbb E_{\tau\sim\pi_\theta}[G_0]$

Here $\tau$ is a trajectory generated by following the policy, and $G_0$ is the return from the start of that trajectory. Training changes $\theta$ so trajectories sampled from $\pi_\theta$ tend to receive higher return.

ℹ

Why Direct Policy Learning Can Feel Natural

If the action is continuous, such as a steering angle, scoring every possible action is awkward. A policy can directly output a useful action distribution instead of enumerating an enormous action list.

Checkpoint

What does $\pi_\theta(a\mid s)$ mean in a policy-gradient method?

←PreviousInstability and OverestimationValue-Based Deep RL Next→REINFORCE IntuitionPolicy Gradients and PPO