Evaluation Protocols
An evaluation protocol defines which checkpoint is evaluated, in which environments, with which seeds, under which policy mode, for how many episodes, with which metrics, and under what pass/fail gates. Without a protocol, a team can accidentally tune the evaluation until the answer looks good.
Training-time reward curves give incomplete deployment evidence. They are collected during learning, often under exploration noise, changing policies, and training environments. Deployment decisions need frozen, held-out, repeated evaluation of the exact policy that might ship.
A strong RL evaluation separates at least four questions:
Evaluation dimensions
Four questions for RL evaluation
A strong evaluation keeps each question visible, because a policy can pass one dimension and fail another.
Task performance
Does the policy achieve the real objective beyond high training reward?
Signals to track
Success rate, real-objective score, goal completion, user outcome.
Robustness
Does it work across held-out seeds, layouts, users, dynamics, prompts, or market conditions?
Signals to track
Held-out scenarios, stress tests, lower-tail return, distribution shift.
Safety and constraints
How often does it collide, violate a budget, over-refuse, under-refuse, exceed latency, or enter an unsafe state?
Signals to track
Violation rate, latency failures, budget breaches, unsafe-state counts.
Recoverability
When something goes wrong, does the system return to safe behavior, escalate, or keep compounding errors?
Signals to track
Fallback success, escalation rate, recovery time, repeated-failure loops.
Evaluation protocol
Build the release gate
Toggle evaluation requirements. The score determines whether the policy is ready for real users, hardware, money, or safety-critical context.
readiness
54%
decision
Block deployment
Evaluation should also include behavioral artifacts. Loss curves rarely explain RL failures by themselves. Useful debugging artifacts include rollout videos, state-visitation heatmaps, action histograms, reward-component plots, value-estimate calibration, and failure galleries.
For real-world deployment, add operational tests. A robot control loop may require inference under a latency budget such as ms. A recommender may need budget and fairness constraints. An RLHF assistant may need refusal appropriateness, hallucination audits, and jailbreak tests. A trading agent may need market-regime stress tests and hard loss limits.
Most RL failures show up in behavior. Loss in RL differs from supervised predictive loss, so a smooth loss curve can still hide a bad policy. Watch for critic loss decreasing as returns decrease, actor entropy collapse, action histogram degeneracy, Q-values diverging from Monte Carlo return estimates, and visitation-count spikes in strange states.
A useful monitoring dashboard should track episode length, return distributions across seeds, policy entropy, action distribution drift, state-visitation heatmaps, value estimate bias, reward components, and constraint violations. If you cannot see the agent behave, you cannot debug it.
Reward Hacking Detection
Watch for returns increasing as true task success stays flat, high reward with low goal achievement, degenerate repetitive behavior, action-distribution collapse, visitation spikes in strange states, or constraint violations rising. These are signs the agent optimized the visible metric and missed the mission.
Which evaluation protocol is strongest for a policy that may be deployed?