State-Value and Action-Value Functions

Once returns are defined, we can ask how good a situation is before the episode is over. In a maze, some positions make high future return more likely even before the agent reaches the goal.

A policy is the agent's decision rule. We write $\pi(a\mid s)$ for the probability of taking action $a$ in state $s$. A deterministic policy puts all probability on one action. A stochastic policy spreads probability across actions.

The behavior we want to optimize is the policy. The optimal policy is written $$\pi^*=\arg\max_\pi \mathbb{E}_\pi[G_t]$$ In words: find the policy that maximizes expected return.

A state-value function evaluates being in a state and then following a policy:

$$V^\pi(s)=\mathbb{E}_\pi[G_t\mid S_t=s]$$

Read this as: the value of state $s$ under policy $\pi$ is the expected return, assuming the agent starts in state $s$ at time $t$ and follows $\pi$ afterward. The expectation is necessary because the policy may be stochastic and the environment may be uncertain.

An action-value function evaluates one first action and then following the policy:

$$Q^\pi(s,a)=\mathbb{E}_\pi[G_t\mid S_t=s,A_t=a]$$

Read this as: how good it is to be in state $s$, take action $a$ now, and then follow policy $\pi$ afterward. The difference is practical: $Q$ answers "what if I take this action first?" while $V$ answers "how good is this state if I behave according to this policy?"

The two $Q$ values are action-specific: shortcut has one value, safe path has another. The $V$ value is the average return from the state if the policy chooses actions with its usual probabilities.

That is why $V$ is policy-dependent. The action-values can stay fixed while the state-value changes, because a different policy changes how often each action is chosen.

The relationship is $$V^\pi(s)=\sum_a \pi(a\mid s)Q^\pi(s,a)$$ This is a weighted average. Each $Q^\pi(s,a)$ is counted in proportion to how often policy $\pi$ takes action $a$ in state $s$.