Environment and Reward Contracts

In supervised learning, the dataset is usually a file, table, or corpus that can be versioned directly. In RL, the "dataset" is generated by an environment. If the environment changes, the stream of experience changes, even if the algorithm and random seed stay fixed.

The transition model defines the dynamics:

$s_{t+1}\sim P(\cdot\mid s_t,a_t)$

If $P$ changes, the learning problem changes. A simulator upgrade, a map edit, an action-scaling wrapper, or a new termination condition can create a different training distribution under the same high-level task name.

Environment pinning

Pin the data generator

A reproducible RL report should pin the environment at several levels.

pinned areas

0/7

readiness

environment

Gaps remain

If the environment changes, the stream of experience changes with it.

⚠

Rewards Are Code

Changing reward code changes the optimization objective. If $r_t=R(s_t,a_t,s_{t+1})$ changes, then the optimal policy can change too. A higher return under a modified reward needs a same-evaluator comparison before it can support an algorithm claim.

Reward engineering

Lock and inspect the reward

Go down the checklist to make the reward objective versioned, transparent, and testable.

checked items

0/8

readiness

reward record

Gaps remain

Component logging and reward tests help reveal policies that improve the score for the wrong reason.

ℹ

Golden Trajectory Tests

A golden trajectory test is a regression test for environment, wrappers, reward, reset logic, and stepping. Fix an environment version, seed, and simple scripted policy; run for $T$ steps; record $\tau=(s_0,a_0,r_0,s_1,\ldots,s_T)$ ; then assert that future runs match the saved reference within tolerance.

Exact equality is often too strict for floating point systems. Better golden tests may assert that reward is within tolerance, episode length matches, final position is within tolerance, cumulative reward is within tolerance, and key event sequences match. Key events might include collisions, pickups, constraint violations, goal reached, timeout, or fallback activation.

Essentially, golden trajectory tests are unit tests for an interactive system. They verify that the world the policy trains in has stayed stable.

Checkpoint

Why should reward components be logged separately from total return?

←PreviousComparing Runs FairlyReinforcement Learning Engineering Next→Distributed RL and DeploymentReinforcement Learning Engineering