Introduction to Modern Reinforcement Learning

An option is a reusable behavior that the agent can choose as if it were an action. The option then runs its own lower-level policy until it decides to stop. In a robot domain, options might include "navigate to doorway," "dock at charger," "open drawer," or "grasp visible object." In a software agent domain, they might include "search the database," "draft a reply," or "verify a transaction."

Skills and options are useful when they are reusable across states and tasks. A door-opening skill is more valuable if it works for many doors, not just the one door seen during training. A navigation option is more valuable if it can be selected by many high-level goals, not only by the exact route that created it.

Options framework

Assemble an option tuple

Build a reusable skill by setting when it can start, what it does internally, and when it should stop.

Initiation set

Near doorwayBattery lowCarrying tote

Internal policyTermination confidence

Initiation rules

1

Termination

80%

Safety margin

82%

Option card

I_o: near doorway

pi_o: navigate through the nearest doorway, slowing near people

beta_o: stop when completion confidence reaches 80%.

This option is selectable and reasonably reusable.

An option is useful because the high-level policy can select it as one action while the option manages many primitive steps.

The classic options framework defines an option as a triple:

$o=\left(I_o,\pi_o,\beta_o\right)$

$I_o$ is the initiation set: the states where the option is allowed to start. $\pi_o$ is the intra-option policy: the behavior that chooses primitive actions while the option is active. $\beta_o(s)$ is the termination condition: the probability or rule that ends the option in state $s$ .

This formal structure is practical. If $I_o$ is too broad, the skill may start in states where it cannot succeed. If $\beta_o$ is too weak, the skill may keep running after it has failed or after its goal is already complete. If $\pi_o$ overfits one context, the skill will not transfer.

Equation walkthrough

An option is a macro-action

Click each piece of the option tuple to see how a skill becomes selectable, executable, and stoppable.

includes

Option

one macro-action

An option omega is a multi-step behavior the high-level policy can choose. Once selected, it runs its own policy until its termination rule stops it.

Options let the agent reason at a higher time scale: choose a skill once, then let it handle many primitive steps.

When a high-level policy chooses an option, the learning problem becomes a semi-Markov decision process because actions can last variable amounts of time. A Q-learning-style update for an option can include the option duration $\tau$ :

$Q(s,o)\leftarrow Q(s,o)+\alpha\left[r_{t:t+\tau}+\gamma^\tau\max_{o'}Q(s',o')-Q(s,o)\right]$

The $\gamma^\tau$ term matters because a longer option delays the next high-level decision. The manager should prefer a long option only when the reward and resulting state justify giving up that control for several steps.

Equation walkthrough

Updating the value of an option

Click each term to see how Q-learning changes when one high-level option can run for several primitive steps.

\leftarrow

+[+]

Reward during the option

all reward collected while option o runs

Because an option lasts $\tau$ steps, the target includes the reward accumulated from time $t$ until the option terminates.

Option target

$r_{t:t+\tau}+\gamma^\tau\max_{o'}Q(s',o')$

TD error

$\text{target}-Q(s,o)$

Value update

$\text{old value}+\alpha\cdot\text{TD error}$

This is a semi-Markov update: the manager learns from the whole option execution, then jumps to the next high-level decision point.

Giving the agent options raises the next question: which skills should it learn, and when should each skill stop? Option-Critic learns these pieces automatically. It learns the option policies $\pi_\omega$ , the termination functions $\beta_\omega$ , and the policy over options $\pi_\Omega$ .

The option-policy gradient updates how an option behaves while it is active. The termination gradient updates whether the option should stop in a state. If the option is doing well, the agent should push the stop probability down. If the option is doing poorly, it should push the stop probability up.

Equation walkthrough

Option-Critic learns skills and stops

Select a gradient to see whether Option-Critic is updating the behavior inside an option or the decision to terminate it.

Option policy gradient

update how an option acts

This is a standard policy-gradient update inside a chosen option. It pushes the option policy toward actions with positive advantage and away from actions with negative advantage.

\theta

Option-policy parameters

The parameters controlling the intra-option policy pi_omega.

\pi_\omega(a\mid s)

Intra-option policy

The action distribution used while option omega is active.

A_\omega(s,a)

Option advantage

How much better action a was than the option's usual behavior in state s.

\nabla_\theta\log\pi_\omega(a\mid s)

Probability push

The term that increases or decreases the probability of the sampled action.

Option-Critic learns option policies, termination functions, and the policy over options, but the learned options can still collapse without useful structure.

⚠

Option Collapse

In practice, Option-Critic can learn options that immediately terminate, never terminate, behave like primitive actions, or fail to generalize. The agent has no built-in grounding for what a useful human-style skill should be.

ℹ

What Makes a Skill Reusable?

A reusable skill has a clear initiation set, a policy that solves a meaningful subproblem, a reliable termination rule, and behavior that depends on task-relevant state rather than incidental training details.

Options and Skills

Assemble an option tuple

An option is a macro-action

Updating the value of an option

Option-Critic learns skills and stops