Seeds and Reproducibility

RL is unusually sensitive to randomness. Initialization, exploration, environment stochasticity, replay-buffer ordering, data-loader behavior, action sampling, distributed actor timing, and GPU nondeterminism can all change the trajectory of learning. Two runs can use the same code, hyperparameters, environment, and reward function, yet different random seeds can lead to very different final policies: one policy may solve the task reliably, while another may fail or learn unsafe behavior.

That is why deployment teams should care about the full training process. If retraining the same system next week could produce a much worse policy, that instability is itself a product risk.

Different seeds can lead to different learned policies, even when the training script "converges":

$$\hat{\pi}_1\neq\hat{\pi}_2\quad\text{and}\quad J(\hat{\pi}_1)-J(\hat{\pi}_2)\text{ can be large}$$

So the reproducibility question has two parts: "can I rerun this seed?" and "does this method produce acceptable policies across seeds?"

Reproducibility has two layers:

• Exact replay: can this run be reproduced closely enough to debug it?
• Statistical reliability: does the method work across seeds?

Exact replay is hard in physics simulators because multithreading, floating point differences, variable timesteps, and contact-resolution order can cause trajectories to diverge. The practical target is often "deterministic enough" for unit tests and auditable enough to explain divergence.

When reporting an RL result, summarize the distribution across seeds. A basic report should include mean return, variance or confidence intervals, lower-tail performance, failure rate, and constraint violations.

For deployment, the lower tail matters. A warehouse robot policy with seven reliable seeds and one seed that collides repeatedly reveals that the training recipe can produce an unacceptable policy unless selection, evaluation, and safety gates catch it.

Many simulators add another wrinkle: even with fixed seeds, stepping can be nondeterministic. Causes include multithread scheduling, non-fixed integration timesteps, floating point differences across hardware, GPU physics, and nondeterministic collision resolution order. If multiple contacts are resolved in a different order, the resulting forces and positions can differ, and that difference can compound over a long rollout.

Simulator-dependent knobs that can make a run more deterministic include single-threaded simulation, fixed timesteps, deterministic solvers or contact ordering when supported, and disabling GPU physics. Physics-heavy RL often cannot reach full determinism. The practical goal is to be deterministic enough for regression tests and to measure replay divergence: how quickly two supposedly identical trajectories drift apart.