Open Research Challenges
Hierarchical RL is appealing because it matches how people describe long-horizon behavior: break a hard task into skills, compose the skills, reuse them later. The hard part is making that story work reliably in learned systems. A hierarchy can help, but it can also hide brittle skills, introduce extra objectives, and make evaluation less fair.
Research triage
Drag each failure into the right challenge
Hierarchical RL problems often sound alike. Sort the cases by the specific bottleneck they reveal.
The manager blames the navigation skill for a failure that was actually caused by a bad pickup skill.
Drag me
A hierarchy beats a flat policy only because it has three times more parameters and training budget.
Drag me
A robot invents a beautiful spinning skill that maximizes novelty but never helps deliveries.
Drag me
A door-opening skill works in one building but fails when handles, hinges, and approach angles change.
Drag me
Discovery
Transfer
Credit
Evaluation
Placed
0/4
Matched
0/4
Evidence quality
0%
Sort each case by the research problem that most directly explains the failure.
Skill discovery asks where options and subgoals come from. Random novelty is not enough; useful skills should compress future learning or solve repeated bottlenecks.
Transfer asks whether a skill remains useful when objects, layouts, dynamics, or goals change.
Credit assignment asks which layer deserves praise or blame when behavior succeeds or fails.
Evaluation asks whether the hierarchy itself caused the improvement.
Evidence That Hierarchy Helped
Look for faster learning on long-horizon tasks, better transfer to new goals, improved exploration of sparse-reward environments, interpretable reuse of skills, and ablations showing that the hierarchy matters beyond model size or compute.
Evaluation Trap
Do not compare a carefully engineered hierarchical system against an under-tuned flat baseline and conclude that hierarchy is the reason for the improvement. Fair comparisons should control for architecture, data, compute, reward shaping, and privileged task knowledge as much as possible.
What evidence best supports the claim that hierarchy improved learning rather than merely adding complexity?