Introduction to Modern Reinforcement Learning

REINFORCE is the simplest policy-gradient idea: raise the log-probability of actions that preceded good outcomes, and lower the log-probability of actions that preceded bad outcomes.

Imagine a chef trying a new seasoning. If the meal receives a high rating, the chef becomes more likely to use that seasoning again in a similar situation. If the rating is poor, the chef becomes less likely to repeat it. REINFORCE does the same thing with sampled actions and returns.

REINFORCE

Good outcomes raise the probability of sampled actions

The sampled action was move right. Choose the return from the trajectory and watch what happens to that action's probability.

\nabla_\theta J(\theta) = \mathbb E\left[\sum_t \nabla_\theta \log \pi_\theta(a_t\mid s_t)G_t\right]

Return G_t

+6

Update

raise

Chosen action

right

π_old

40%

π_new

54%

The sampled action preceded a good outcome, so REINFORCE nudges its log-probability up.

REINFORCE is simple, but the return G_t is a noisy sample of what happened after the action.

The policy-gradient identity is often written as

The equation below is rendered by the same KaTeX helper used inside the interactive, so the symbols stay readable while the surrounding text explains the intuition.

Equation walkthrough

The REINFORCE update signal

=[]

Probability direction

which way raises this action

This points in the parameter direction that would make the sampled action more likely in this state.

Pieces

The term $\log \pi_\theta(a_t\mid s_t)$ is the log-probability of the action the agent actually sampled. The return $G_t$ is the discounted sum of future rewards from time $t$ onward. If $G_t$ is large and positive, the update pushes the sampled action's probability up. If $G_t$ is poor, it pushes the probability down.

⚠

A Return Is Only One Sample

$G_t$ is what happened on one collected trajectory. It is a Monte Carlo sample of $Q^\pi(s_t,a_t)$ , not the exact expectation over every possible future. That is why REINFORCE can have high variance.

Sample vs expectation

One collected trajectory is not the whole value

Hold fixed the same state s_t and first action a_t. Choose which collected training trajectory actually happened.

start at states_t+take actiona_tthen future randomness unfolds underπ

G_t what actually happened

+4

The robot gets there, but takes extra steps after the same first action.

A single sample from the collected training trajectories. Another rollout from the same starting decision could have produced a different return.

Q^π(s_t, a_t) expectation

2.0

an expectation over all future randomness under the policy

G_t is one observed rollout. Q^π(s_t, a_t) is the average over all futures after taking a_t and then following π.

The problem is that $G_t$ can be huge and noisy. One rollout might get lucky; another might begin in a much worse situation. High variance makes learning unstable because the policy is pushed around by raw scores that mix two questions together:

Was this a good situation overall?
Did my particular action help or hurt?

Policy learning mostly cares about the second question. This is where a baseline becomes useful. Grading on a curve means we do not care about the raw score by itself; we care whether the outcome was better or worse than expected for that state.

Variance control

Grade the return on a state-specific curve

Raw returns can be huge and noisy. A baseline subtracts the typical outcome for the state before judging the sampled action.

Score components

Raw return G_t

+11

Typical outcome b(s_t)

+9

Curved score G_t - b(s_t)

+2

Learning signal

+2

The update asks: was this outcome better or worse than normal for this state?

The raw return is huge, but much of that score came from starting in an easy situation.

Raw-return spread

15

After-baseline spread

6

Subtracting a baseline b(s_t) lowers variance without changing the expected policy-gradient direction, as long as the baseline does not depend on the sampled action.

So we replace the raw return $G_t$ with a centered return:

$G_t-b(s_t)$

The baseline $b(s_t)$ subtracts the typical outcome for the state. If the baseline depends only on the state, not on the sampled action, it lowers variance without changing the expected policy-gradient direction:

$\nabla_\theta J=\mathbb E\left[\sum_t\nabla_\theta\log\pi_\theta(a_t\mid s_t)(G_t-b(s_t))\right]$

A natural choice is $b(s_t)\approx V^\pi(s_t)$ , because $V^\pi(s_t)$ is a running forecast of the future: when I am in this situation, how well do things usually turn out if I behave normally?

This matters because $Q^\pi(s_t,a_t)$ mixes two things together: how good the state already is, and how much the particular action helps or hurts. Subtracting $V^\pi(s_t)$ removes the first part, leaving the action-specific residual.

Advantage derivation

From noisy returns to better-than-expected actions

Step through why choosing the value function as the baseline leaves the action-specific advantage behind.

b(s_t)\approx V^\pi(s_t)

The value is a running forecast: when I am in this situation, how well do things usually turn out if I behave normally?

Before subtracting

G_t carries both the situation and the action.

V^π(s_t) good or bad state

A^π(s_t, a_t) action effect

-

After subtracting the value baseline

G_t - V^π(s_t) keeps the action effect and drops the state-quality part.

That leftover is the advantage signal.

Review: split value from action advantage

Q(s, a)=V(s)+A(s, a)

Dueling networks use a centered advantage term, but the idea here is simpler: separate how good the state is from what this action adds.

V(s) value stream

How good this state is overall. It controls for the situation.

A(s, a) advantage stream

How much better this action is compared to the other actions available here.

V^π(s) removes the luck of being in a good or terrible state, so actions are judged more fairly.

Choosing $b(s_t)\approx V^\pi(s_t)$ is the bridge to advantage. The value baseline controls for the situation; the remaining signal is the advantage stream, the part that says whether the sampled action was better or worse than the other actions available in that state.

Since $G_t$ is a Monte Carlo sample of $Q^\pi(s_t,a_t)$ , the centered return approximates the action's advantage:

$G_t-V^\pi(s_t)\approx Q^\pi(s_t,a_t)-V^\pi(s_t)=A^\pi(s_t,a_t)$

Remember the question the advantage asks: was this action better or worse than expected?

That gives the advantage-weighted policy-gradient update:

$\nabla_\theta J=\mathbb E\left[\sum_t\nabla_\theta\log\pi_\theta(a_t\mid s_t)A^\pi(s_t,a_t)\right]$

Equation walkthrough

The advantage-weighted policy update

Click a term to see its job in the final advantage-weighted update.

=[]

Advantage weight

raise, lower, or ignore

The advantage decides whether to follow that direction. Positive advantage raises the action probability; negative advantage pushes the other way.

Terms

The important change from vanilla REINFORCE is the final multiplier: raw return has been replaced by A^π(s_t, a_t), so the update focuses on whether the sampled action beat the state's normal outcome.