SAC

PPO and TRPO optimize for expected return, but a policy that chases return too aggressively can get overconfident and stop exploring. Soft Actor-Critic, or SAC, changes the learning problem: the agent is rewarded not only for high return, but also for keeping its action distribution usefully stochastic.

SAC adds an entropy term to the objective. Entropy encourages diverse actions, so exploration becomes part of what the agent is trying to optimize instead of something bolted on from outside.

Objective function

$$J(\pi)=\mathbb E_{s_t,a_t\sim\pi}[r(s_t,a_t)+\alpha H(\pi(\cdot\mid s_t))]$$

Here $H(\pi)$ is the entropy of the policy, and $\alpha$ controls the exploration-exploitation tradeoff. High $\alpha$ means the agent values exploration more. Low $\alpha$ lets the policy become greedier and more deterministic.

The critic in SAC learns a soft Bellman function:

$$Q^\pi(s_t,a_t)=r(s_t,a_t)+\gamma\mathbb E_{s_{t+1}\sim P,a_{t+1}\sim\pi}[Q^\pi(s_{t+1},a_{t+1})-\alpha\log\pi(a_{t+1}\mid s_{t+1})]$$

Subtracting $\alpha\log\pi$ encourages policies that keep options open. Instead of maximizing just return, the critic values return minus overconfidence. This soft backup averages over possible future actions, which tends to create smoother value landscapes and easier optimization.

SAC is still an actor-critic method, but the moving pieces are richer:

Actor policy: outputs stochastic actions.

Critic 1 and Critic 2: estimate soft Q-values. Double critics reduce overestimation.

Target networks: provide stable learning targets.

Replay buffer: stores past experiences for off-policy updates.

Instead of fixing $\alpha$, SAC can learn it automatically. The goal is to keep entropy near a target level $H_{target}$.

Temperature loss

$$L(\alpha)=\mathbb E_{a_t\sim\pi_t}[-\alpha(\log\pi_t(a_t\mid s_t)+H_{target})]$$

If the agent is too predictable, $\alpha$ increases, pushing the policy toward more exploration. If the agent is too random, $\alpha$ decreases, allowing more exploitation.

PPO learns from what it just did. SAC learns from what it remembers.

PPO is on-policy: it assumes the trajectories were generated by the current policy. SAC is off-policy: it can safely learn from past policies' experiences stored in a replay buffer.

What does soft mean?

Soft means we do not assume the agent always takes the single best action. Instead, we average over good actions and reward uncertainty. The policy is expected to act well on average, while still preserving randomness where it is useful.