Introduction to Modern Reinforcement Learning

If a run looks surprising three months from now, can you reconstruct what produced it? In supervised learning, a run is usually tied to a fixed dataset and a training script. In RL, the agent creates its own data through interaction, so the environment, reward function, wrappers, seeds, policy checkpoints, and evaluator all become part of the experiment.

Engineering contrast

Why RL runs need more provenance

Supervised ML usually optimizes against a fixed record. RL learns from a moving stream that the policy helps create.

Supervised ML

A model learns from examples that were collected before training.

versus

RL

A policy learns from consequences produced by its own actions.

Fixed dataset

Works from a fixed dataset.

Data source

Self-generated data

The agent generates its own data by acting in the environment.

IID samples

Often assumes independent and identically distributed samples.

Sample structure

Non-IID trajectories

Receives trajectories where each action changes future data.

Predictive loss

Optimizes predictive loss against labels.

Optimization target

Behavioral return

Optimizes behavior under an environment and reward function.

Local update

A bad batch usually affects a local gradient step.

Failure footprint

Compounding path

A bad decision can change the states, rewards, and failures seen later.

The engineering consequence

In RL, a code, seed, wrapper, reward, or simulator change can alter both what the agent learns from and what it is optimizing toward.

In RL, the environment and reward function are first-class parts of the experiment.

In RL, the optimization signal depends on long-term consequences. The environment plus reward function defines the learning problem, so tracking them is as important as tracking the model code.

Run record checklist

Production RL logging

An RL run needs enough provenance to rebuild, compare, debug, and defend the result later.

logged areas

0/6

readiness

0%

run record

Gaps remain

A missing environment, reward, seed, or artifact record can make an impressive policy impossible to explain.

Experiment forensics

Is this the same experiment?

Pick a run-pair. The mean return changed, and the deployment question is whether the evidence is comparable.

Field	Run A	Run B
Algorithm	PPO	PPO
Environment	WarehouseNav-v3	WarehouseNav-v3
Reward	alive + goal	alive + goal - energy
Runtime	sim==1.4.2	sim==1.4.2
Seeds	10 seeds	10 seeds
Code state	clean commit	clean commit

Comparable

no

Audit risk

86%

Deployment action

freeze first

Comparison invalid as a pure algorithm result. The optimization target changed.

Report reward components separately and evaluate both policies under the same frozen evaluator.

A production RL result needs a trail: code, config, environment, reward, seeds, artifacts, and evaluator.

The environment is part of the dataset generator. If the transition dynamics change, the training data distribution changes:

$s_{t+1}\sim P(\cdot\mid s_t,a_t)$

The reward is also code. If $r_t=R(s_t,a_t,s_{t+1})$ changes, the optimization objective changes:

$\pi^*=\arg\max_\pi\mathbb{E}\left[\sum_t \gamma^t r_t\right]$

An improved return under a new reward function shows that the scoring rule changed, so algorithm claims need a frozen evaluator.

⚠

Golden Trajectory Tests

A golden trajectory test fixes an environment version, seed, and simple policy, then checks that the resulting trajectory still matches a saved reference within tolerance. It catches accidental changes to reset logic, wrappers, reward code, physics stepping, and asset files before those changes contaminate experiment comparisons.

Experiment Tracking

Why RL runs need more provenance

Production RL logging

Is this the same experiment?