Subgoals

Subgoals divide a difficult task into intermediate achievements. Instead of learning "solve the whole maze," an agent can learn "reach the doorway," then "cross the hallway," then "enter the goal room." Instead of learning "assemble the product," a robot can learn "grasp the part," "align it," "insert it," and "verify the fit."

Good subgoals help with exploration because they create reachable stepping stones. They help with credit assignment because progress can be evaluated before the final outcome. They help planning because the agent can reason over meaningful waypoints rather than every primitive action.

Subgoals

Place waypoints, then read the route

Tap up to four cells to create intermediate subgoals for a robot crossing a blocked room.

Subgoals

Route length

Alignment

100%

Route script

0-4 -> 0-2 -> 2-2 -> 4-0

These subgoals keep the route direct while moving steadily toward the goal.

Good subgoals are stepping stones: reachable, relevant, and aligned with the original task.

The waypoint exercise shows that subgoals are not automatically helpful. A subgoal can be reachable and still harmful if it pulls the agent away from the real task. A good subgoal is usually close to a bottleneck, constraint, or reusable state: doorways in navigation, grasp poses in manipulation, verification points in software workflows, or stabilization states in control.

One common approach is to learn goal-conditioned policies.

Subgoal hierarchy

High-level goals, low-level actions

Choose a subgoal and watch the hierarchy split the decision: the high-level policy chooses what to pursue, then the low-level policy chooses primitive actions conditioned on that goal.

High-level policy

$g_t\sim\pi_{\text{high}}(g\mid s_t)$

From state $s_t$ = maze entrance, choose subgoal $g_t$ = reach doorway.

Low-level policy

$a_t\sim\pi_{\text{low}}(a\mid s_t,g_t)$

Conditioned on $s_t$ and $g_t$ , choose primitive action $a_t$ = move north.

Low-level rollout

move north

->turn right

->cross threshold

The low-level controller acts until the doorway subgoal is reached. Then the high-level policy can sample a new $g_{t+k}$ .

The low-level controller does not act from state alone. It acts from state plus goal, so the same state can imply different actions under different subgoals.

Subgoals can be hand-designed, learned from demonstrations, discovered from bottleneck states, or generated by representation learning. Hand-designed subgoals are interpretable but may miss clever strategies. Learned subgoals can adapt to data but may exploit reward loopholes. Demonstration-derived subgoals are grounded in human behavior but may inherit human inefficiencies.

The most important test is whether the subgoal improves the original task under realistic conditions. If an autonomous vehicle learns "center in lane" as a subgoal, that is useful only when it remains subordinate to the true objective: safe, legal, comfortable travel. In a construction robot, "pick up tool" is useful only if it leads to completing the work safely, not if the robot repeatedly picks up tools for easy intrinsic reward.

⚠

Subgoal Misalignment

A poorly chosen subgoal can become a local objective that competes with the task. The agent may become very good at reaching intermediate states that are easy to reward but irrelevant, unsafe, or actively counterproductive.

ℹ

Design Check

Ask whether each subgoal is reachable, measurable, reusable, recoverable from failure, and causally connected to the final objective. If a subgoal fails any of those tests, it may add complexity without improving learning.

Checkpoint

When can a subgoal make the original task harder?

←PreviousOptions and SkillsHierarchical RL Next→HIROHierarchical RL