TRPO

In vanilla policy gradient methods, like REINFORCE, each update step moves the policy parameters in proportion to the advantage of sampled actions. Helpful actions get pushed up. Harmful actions get pushed down.

But policy networks are highly nonlinear. A small step in parameter space can cause a huge change in action probabilities. That can collapse exploration or erase prior good behavior.

For example, if a few random trajectories get high rewards, the model can overfit to them and forget everything else it had learned.

The ratio $r_t(\theta)$ compares the probability of the sampled action under the new policy to its probability under the old policy.

If $r_t(\theta)>1$, the new policy assigns higher probability to that action. If $r_t(\theta)<1$, it assigns lower probability.

TRPO's constrained objective is

$$\max_\theta \mathbb E_t[r_t(\theta)A_t]$$

subject to

$$\mathbb E_t[D_{KL}(\pi_{\theta_{old}}\|\pi_\theta)]\le \delta$$

where

$$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{old}}(a_t\mid s_t)}$$

The objective $\mathbb E_t[r_t(\theta)A_t]$ encourages the policy to increase the probability of actions that led to above-average returns and decrease the probability of actions that performed worse than expected.

The KL term $D_{KL}(\pi_{\theta_{old}}\|\pi_\theta)$ measures how different the new policy's action probabilities are from the old policy's probabilities. The constraint $\mathbb E_t[D_{KL}]\le\delta$ keeps the update within a trust region, preventing large destructive policy changes.

That is the central promise of TRPO: steady, stable improvement without forgetting previously learned behaviors.