Planning vs. Learning

Bellman equations describe what values should satisfy when we know the environment. If the agent has the transition model $P(s'\mid s,a)$ and reward model $R(s,a,s')$ , it can plan: reason through possible next states before acting.

In many real problems, the agent lacks those equations. It only has experience: "I was in state $s_t$ , took action $a_t$ , received reward $r_{t+1}$ , and landed in state $s_{t+1}$ ." Learning means using those samples to improve behavior.

Planning vs learning

What information does the agent have?

Choose the information available to the agent. The method follows from that information.

You only see samples: state, action, reward, next state.

Method

Learning

Update Q-values from observed transitions when P and R are unavailable.

Planning uses a known model. Q-learning is model-free: it learns action-values directly from experience samples.

Tabular RL studies this in small finite worlds where values can be stored directly in a table. A Q-table has one entry for each state-action pair. Instead of learning a neural network, the agent updates table entries such as $Q(s,\text{right})$ and $Q(s,\text{up})$ .

Q-learning is model-free: the agent can learn without knowing $P$ or $R$ ahead of time. It uses sampled transitions to learn an estimate of $Q^*(s,a)$ , the optimal action-value function.

ℹ

What Tabular Means

Tabular describes the representation size: the state and action spaces are small enough that the agent can store a separate number for every $(s,a)$ pair.

Checkpoint

Why is Q-learning useful when the agent lacks $P(s'\mid s,a)$ and $R(s,a,s')$ ?

←PreviousBellman EquationsMDPs, Rewards, and Value Functions Next→Exploration vs. ExploitationTabular RL and Q-Learning