Agent-Environment Loop

Supervised learning and reinforcement learning have different shapes. In supervised learning, training data and labels produce a loss, and the loss updates model parameters. In reinforcement learning, the diagram is circular: the agent observes state, takes action, receives reward, and returns to a changed environment.

This loop is the conceptual anchor for the whole course. The agent participates in the process that creates the dataset it learns from.

Comparison of supervised learning and the reinforcement learning loop. — Supervised-learning pipeline beside the reinforcement-learning loop.

Agent-environment loop

Trace the loop by hovering over each part

Hover or tap the diagram labels to define the pieces of the reinforcement learning loop.

Hover definition

Agent

The decision-making system. It contains a policy and may contain a learning algorithm that updates that policy.

Example

A game player, robot, recommender, or language-model system choosing what to do next.

The loop is circular: the policy chooses an action, the environment returns reward plus a changed state, and the RL algorithm uses that experience to update the policy.

A single loop iteration can be written as a transition:

$S_t \xrightarrow{A_t} (R_{t+1}, S_{t+1})$

The policy, usually written $\pi$ , determines how actions are chosen. Sometimes the policy is deterministic: in this state, take this action. Often it is stochastic: in this state, choose among actions with certain probabilities.

RL aims to learn behavior that performs well over time. That is why later chapters introduce return, discounting, value functions, and Bellman equations.

⚠

The Loop Creates Feedback Problems

If the current policy never visits a useful part of the environment, the agent may never collect the evidence needed to discover that better behavior exists. Exploration is part of learning.

Checkpoint

In one RL transition, what does the next reward represent?

←PreviousWhy Reinforcement Learning MattersFrom AlphaZero to RLHF Next→What Is an Agent?From AlphaZero to RLHF