Introduction to Modern Reinforcement Learning

Proximal Policy Optimization, or PPO, inherits TRPO's principle: improve performance without drifting too far. But instead of solving a KL-constrained optimization problem, PPO directly flattens the objective when updates get too large.

That flattening prevents the policy from over-correcting on a single batch. Conceptually, clipping means: do not get too excited about one lucky game.

Equation walkthrough

PPO flattens the objective when updates get too large

Click each term to see how the clipped surrogate objective mimics TRPO's trust-region idea without solving a constrained optimization problem.

=[(,)]

Clip operation

flatten excessive changes

Clipping limits how far the probability ratio can move away from 1, flattening the objective once the update becomes too large.

Terms

Clipping means: do not get too excited about one lucky game. Large probability changes stop earning extra objective gain.

The clipped PPO objective is

$L^{CLIP}(\theta)=\mathbb E_t\left[\min\left(r_t(\theta)A_t,\operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)A_t\right)\right]$

where

$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{old}}(a_t\mid s_t)}$

If $r_t(\theta)>1$ , the new policy assigns higher probability to that action. If $r_t(\theta)<1$ , it assigns lower probability.

The operation $\operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)$ limits how far the ratio can move away from $1$ , preventing overly large updates. The $\min$ operator takes the smaller, more conservative improvement between the unclipped and clipped objectives.

The parameter $\epsilon$ controls the size of this practical trust region: smaller $\epsilon$ makes learning more cautious, while larger $\epsilon$ allows bolder updates.

Epsilon lab

Epsilon controls how aggressive PPO is allowed to be

Move the new policy probability and the clipping range. PPO lets useful changes through, then flattens the gradient once the probability ratio leaves the safe region.

New probability for the good action72%

Clip range ε0.20

π_old

50%

π_new

72%

r_t(θ)

1.44

safe range

0.80-1.20

unclipped

2.88

used by PPO

2.40

Gradient flattened

The action was good, but PPO gives no extra objective gain for pushing its probability this far on the same batch.

Smaller ε makes learning more conservative. Larger ε allows bolder updates, but the policy can over-correct more easily.

PPO clipping

No extra credit for an oversized probability jump

Keep the old probability fixed at 50% and the advantage fixed at A_t = +2. Choose the new policy's probability for the same sampled action.

What PPO clips

r_t=new probability/old probability

clipr_tto the range1 - epsilonthrough1 + epsilon

use the smaller ofr_t x A_tandclipped r_t x A_t

π_old

50%

π_new

80%

ratio r_t

1.6

clipped ratio

1.2

r_tA_t

3.2

used value

2.4

Clipped: the action was good, but PPO gives no extra credit for making the probability jump too far.

For a positive advantage, PPO rewards increasing the action probability, but clips the gain once the ratio is too far from 1.

The min-plus-clip trick mimics TRPO's KL constraint implicitly. PPO still tries to keep updates within a safe policy region, but it does so with a clipped surrogate objective that is much easier to optimize.

If the new policy increases the probability of a good action just a little, great. If it increases that probability too much, PPO clips the gain: no extra reward for extreme changes.

Unlike REINFORCE or simple A2C updates, PPO can reuse the same freshly collected trajectories for several gradient steps. That improves sample efficiency while staying on-policy enough for stable learning.

ℹ

Why PPO Dominates Modern Pipelines

PPO fits actor-critic architectures easily, handles noisy feedback reasonably well, works with large-batch GPU training, and is comparatively robust to hyperparameters.

That combination matters for later RL pipelines: the method is stable enough to train large policies, simple enough to implement at scale, and flexible enough to tolerate messy advantage estimates.

ℹ

Epsilon as a Tuning Knob

Small $\epsilon$ means conservative and stable. Large $\epsilon$ means more aggressive and exploratory, but riskier. PPO turns a difficult trust-region problem into a simple learning-aggressiveness knob.

In practice, $\epsilon$ decides how much PPO is allowed to trust one batch before it says: enough, no more extra gain from moving farther.