Temporal Abstraction

Humans solve complex tasks in chunks: "make coffee," "get ready for work," or "drive to the airport." Each chunk hides dozens or hundreds of primitive actions. Standard RL agents usually see only primitive actions such as move forward, turn left, pick up object, or open door. That works for simple tasks, but long-horizon problems quickly become painful: the agent can get lost, reward is too delayed, credit assignment explodes, and exploration becomes inefficient.

Hierarchy changes the scale of decision-making. You do not plan "rotate wrist, step forward, adjust grip" when you want the fridge open. You select the macro-action "open the fridge." If an RL agent can choose multi-step behaviors, exploration becomes more structured, decisions happen less frequently, rewards can become more predictable, and credit assignment spans fewer high-level steps.

Hierarchy is not automatically better. If the abstraction is too fine, the agent still faces the original long-horizon problem. Credit assignment is difficult because reward at the end of the episode must be traced through many small actions. Exploration is also inefficient: the agent may spend most of its time rediscovering basic behaviors instead of composing them.

If the abstraction is too coarse, the high-level policy loses flexibility. A long routine like "complete the whole order" may work in familiar conditions, but it can fail badly when an aisle is blocked, a sensor glitches, or a human steps into the workspace. Good temporal abstractions sit between these extremes: long enough to simplify planning, short enough to adapt.

One way to write the hierarchy is:

$$o_t \sim \mu(o\mid s_t), \qquad a_{t+i}\sim\pi_o(a\mid s_{t+i}) \text{ until } \beta_o(s_{t+i})=1$$

The manager policy $\mu$ selects an option or skill $o$. The option's internal policy $\pi_o$ chooses primitive actions. The termination rule $\beta_o$ decides when control returns to the manager. The duration is variable: one option might end after two steps, another after fifty.