Comparing Runs Fairly

Fair comparison means the conclusion follows from the thing you intended to test. If Algorithm B beats Algorithm A, first ask what else changed. In RL, many hidden variables can change the answer: environment version, reward code, wrappers, budgets, seeds, evaluation mode, checkpoint selection, hardware, and rollout throughput.

This matters in real deployment because comparison claims become roadmap decisions. A team may choose a more complex algorithm, ship a policy, buy hardware, expand actor pools, or relax safety review based on a comparison. If the comparison is confounded, the organization makes the wrong engineering bet.

Because RL variance is high, experiments usually need multiple seeds. A practical starting point is $n\ge 5$ to $10$ seeds, confidence intervals over returns, controlled ablations for architecture and reward-shaping changes, and statistical tests over performance distributions when the decision depends on a claimed improvement.

Ablations should change one thing at a time: reward shaping, replay buffer size, network architecture, rollout length, actor count, or exploration schedule. If several knobs change together, the report may still be useful engineering telemetry, though it gives weak evidence for what caused the improvement.

Distributed RL adds another comparison trap. More actors can produce more trajectories per second and can also introduce off-policy drift if actors collect data using stale policy parameters. A faster wall-clock curve may reflect infrastructure gains more than algorithm quality.

Throughput:

$$\text{throughput}\approx \#\text{actors}\times\text{steps per second per actor}$$

A method that wins by using five times more experience should be reported differently from a method that learns more from the same experience budget.