Introduction to Modern Reinforcement Learning

Curiosity-driven learning gives the agent an intrinsic reward for discovering something, even before the task reward appears. This is useful when external reward is rare. A robot might need to learn what a door handle does before it can ever receive reward for reaching the next room.

The hard part is deciding what should count as discovery. Novel pixels, prediction error, state counts, and information gain can all be used as curiosity signals, but they do not point to the same behavior.

Intrinsic reward

Curiosity can help or distract

Pick a curiosity signal for an agent that needs to open a door. One part of the room is useful but unfamiliar; another is noisy forever.

Door

Noisy screen

Goal

Useful exploration40%

Distraction risk88%

Likely target

Noisy screen

Unpredictable noise keeps producing error, so the agent can be rewarded for staring at something irrelevant.

The safest curiosity signals distinguish useful uncertainty from irreducible noise.

A common decomposition writes the learning signal as an external task reward plus an intrinsic bonus:

$r_t^{total}=r_t^{external}+\beta r_t^{intrinsic}$

The coefficient $\beta$ says how much curiosity matters relative to the real task. If $\beta$ is too small, curiosity may not overcome sparse reward. If $\beta$ is too large or poorly defined, the agent may optimize curiosity itself instead of using curiosity to find task-relevant evidence.

Intrinsic reward methods

Choose the signal behind curiosity

Each method turns a different kind of novelty or uncertainty into an intrinsic reward. Click a method to inspect what it measures.

ICM

Intrinsic Curiosity Module

Forward-model prediction error in feature space.

Equation

r_t^i = \left\|\phi(s_{t+1}) - \hat{s}_{t+1}\right\|^2,\quad \hat{s}_{t+1}=f_\eta(\phi(s_t),a_t)

ICM rewards controllable surprise. The model learns a feature representation and a forward dynamics model; high error means the agent reached a transition it does not yet understand. In the original ICM idea, an inverse-dynamics objective helps the features focus on things the agent's actions can affect.

r_t^i

intrinsic reward at time t

\phi(s_t)

learned feature embedding of the current state

\hat{s}_{t+1}

predicted next-state embedding

f_\eta

forward model with parameters eta

a_t

action taken by the policy

\left\|\cdot\right\|^2

squared prediction error, usually using Euclidean distance

1

Encode state

2

Predict next embedding

3

Reward prediction error

The agent encodes the current state, takes an action, predicts the next state embedding, then compares that prediction with what actually happened.

Good fit

Structured environments where the agent can learn controllable transition dynamics.

Watch out

Raw prediction error can chase noise if the representation is too close to unpredictable pixels.

The shared goal is not surprise for its own sake. The bonus should help the agent gather evidence that improves future control.

Random Network Distillation architecture with a frozen random target network and trainable predictor network. — Random Network Distillation compares a trainable predictor against a fixed random target network; high prediction error acts as an intrinsic novelty reward. [Source]

⚠

The Noisy-TV Trap

Prediction error is not always useful information. A source of random noise can remain unpredictable forever, so a prediction-error bonus can reward the agent for watching noise instead of learning controllable structure.

Without noisy TV

With noisy TV

[Source]

Episodic memory

How an episodic curiosity module creates a bonus

Click each component to trace how the current observation becomes an intrinsic reward and, when novel enough, gets stored in memory.

Selected component

Memory buffer

Episode memory

Math handle

M_t=\{e_1,\ldots,e_{t-1}\}

Role in the module

The buffer stores embeddings of states the agent has already visited in the current episode.

The memory module helps the agent avoid repeating the same familiar states. Novelty is judged against what the agent has already visited in this episode.

Curiosity diagnostics

When does curiosity go wrong?

Click a failure mode to connect the visible symptom to the likely cause, fix, and monitor.

Symptom

Return drops when random pixels or irrelevant stochastic events appear.

Likely cause

The bonus is tied to pixel-level unpredictability.

Practical fix

Use feature-space dynamics, episodic kNN gating, and clipped or normalized intrinsic reward.

Monitor

Intrinsic reward distribution and novelty maps near distractor regions.