Actor-Critic Methods

REINFORCE uses raw returns, and raw returns can be noisy. Some trajectories start in lucky states. Others start in difficult states. A high return does not always mean the chosen action was especially good; it may mean the situation was already easy.

Actor-critic methods add a second learner. The actor is the policy $\pi_\theta$ that chooses actions. The critic estimates how good a state usually is, often as $V_w(s)$. The critic acts like a baseline, so the actor can ask a fairer question: was this action better or worse than expected?

The simple actor-critic advantage estimate is

$A_t$ ≈ $G_t - V_w(s_t)$

If $A_t>0$, the sampled action did better than the critic expected for that state, so the actor should make it more likely. If $A_t<0$, the sampled action did worse than expected, so the actor should make it less likely.

This connects back to the value-function chapter: $G_t$ is a sample of $Q^\pi(s_t,a_t)$, and $V^\pi(s_t)$ is the expected return from the state before choosing the particular action. Their difference is the advantage: $A^\pi(s_t,a_t)=Q^\pi(s_t,a_t)-V^\pi(s_t)$.

Was it more difficult to remember today's breakfast or your breakfast two weeks ago? Credit assignment has the same shape. The farther back a consequence stretches, the harder it is to decide which action deserves credit.

Memory decay is controlled by $\lambda$. Higher $\lambda$ means the advantage estimate keeps more context from later timesteps, but it also admits more variance. Lower $\lambda$ means shorter memory, steadier updates, and more bias toward immediate feedback.

In A2C, the critic can give a single snapshot of how good a state is. But sometimes feedback should stretch across multiple timesteps. Generalized Advantage Estimation, or GAE, smooths that feedback by blending short- and long-term memory.

GAE starts with the critic's one-step surprise:

$$\delta_t=r_t+\gamma V(s_{t+1})-V(s_t)$$

Here $V(s_t)$ is what we thought this state was worth before acting, $r_t$ is the actual reward received now, and $\gamma V(s_{t+1})$ is what we expect the future to be worth. Then GAE averages recent surprises, weighted by memory decay:

$$A_t^{GAE}=\delta_t+(\gamma\lambda)\delta_{t+1}+(\gamma\lambda)^2\delta_{t+2}+\cdots$$

$A_t$ measures how much better or worse an action was compared to what the policy expected at that state. If $A_t>0$, the action was better than expected. If $A_t<0$, it was worse than expected.

The tuning question is bias versus variance. As $\lambda\to 1$, GAE captures longer-term consequences, but adds noise. As $\lambda\to 0$, GAE is steadier and lower variance, but more forgetful. GAE lets us blend the two smoothly.