HIRO

HIRO, short for Hierarchical Reinforcement Learning with Off-Policy Correction, makes the subgoal idea more scalable. The agent is split into two policies: a high-level policy chooses a subgoal $g_t$ , and a low-level policy takes primitive actions to reach that goal. Every $k$ steps, the high-level policy updates the goal.

The difficulty is off-policy replay. The replay buffer contains old goals from older versions of the high-level policy, but the low-level policy may have changed. Old data no longer matches new behavior, so the high-level policy can become confused.

HIRO corrects this mismatch by relabeling the stored goal with a goal that explains what actually happened:

$g_t'\approx s_{t+k}-s_t$

In words: what goal would have produced the observed low-level behavior?

Equation walkthrough

HIRO fixes stale goals in replay

Click each equation to see how HIRO separates high-level subgoals from low-level actions.

Goal relabeling

what goal explains the observed behavior?

HIRO replaces stale stored goals with a goal that approximately matches the state change the low-level policy actually produced.

g_t'

Relabeled goal

A corrected goal used for off-policy training.

s_t

Starting state

The state before the low-level policy executed for k steps.

s_{t+k}

Reached state

The state after k low-level steps in the replayed trajectory.

HIRO makes off-policy replay more consistent by asking: what goal would have produced the behavior we actually saw?

HIRO hierarchical reinforcement learning architecture and training design — The design and basic training of HIRO [Source]

Checkpoint

Why does HIRO relabel goals in the replay buffer?

←PreviousSubgoalsHierarchical RL Next→Open Research ChallengesHierarchical RL