PPO Optimization

Given the KL-regularized objective, we want a policy that maximizes reward while staying close to the reference model:

$$\max_\phi \mathbb{E}_{x,y\sim\pi_\phi}\left[r_\theta(x,y)-\lambda D_{KL}(\pi_\phi(\cdot\mid x)\,\Vert\,\pi_{ref}(\cdot\mid x))\right]$$

We cannot enumerate every possible completion, and reward is only defined on sampled trajectories. So RLHF turns the objective into stochastic optimization: sample outputs, estimate reward minus KL, and take gradient steps.

A simple policy-gradient view is:

$$\nabla_\phi J=\mathbb{E}\left[A(x,y)\nabla_\phi\log\pi_\phi(y\mid x)\right]$$

where

$$A(x,y)=r_\theta(x,y)-\lambda D_{KL}-\text{baseline}$$

This is the REINFORCE idea: increase the probability of sampled outputs with positive advantage and decrease the probability of outputs with negative advantage. But for large language models, large reward signals can cause massive policy shifts, KL explosion, reward hacking, or collapse.

PPO stabilizes that update by comparing the current policy to the policy that produced the sampled data:

$$r_t=\frac{\pi_\phi(a\mid s)}{\pi_{old}(a\mid s)}$$

Then it maximizes the clipped surrogate:

$$L_{PPO}=\mathbb{E}\left[\min(r_tA,\operatorname{clip}(r_t,1-\epsilon,1+\epsilon)A)\right]$$

The ratio $r_t$ asks how much more likely the new policy makes the sampled action. The clip prevents one batch from pushing probabilities too far in a single update.

The policy-optimization loop has four conceptual steps:

1. Sample responses from the current policy for a batch of prompts.
2. Score each response with the reward model, plus any safety or KL penalties.
3. Estimate which sampled actions were better than expected.
4. Update the policy carefully so good samples become more likely without causing a large distribution shift.

This is still reinforcement learning. The "environment" is the prompt-and-response process, the action is the generated text, and the reward is learned from human or AI feedback.

GRPO, or Group Relative Policy Optimization, makes the baseline local to the prompt. Instead of asking whether one response has a high score in absolute terms, it samples a small group of responses to the same prompt and asks which ones beat the group's average.

This is useful for language models because reward scales can vary across prompts. A score of 7 might be excellent for a difficult safety prompt but mediocre for an easy summary prompt. The group baseline turns the update into a within-prompt comparison.

A simple group-relative advantage can be written as

$$A_i=r_i-\frac{1}{K}\sum_{j=1}^{K}r_j$$

Each response is compared to the average reward for the sampled group. Responses above the group mean receive positive advantage; responses below it receive negative advantage. This makes the training signal prompt-local: the model asks which sampled answer was better for this prompt, not merely which answer had a high global score.