Safety Constraints and Evaluation

Safe RL treats some outcomes as unacceptable, not merely low reward. A delivery robot should not clip pedestrians to save time. A medical recommender should not explore risky treatments on real patients. A language model should not improve average helpfulness by becoming less reliable on safety-critical slices.

This is why evaluation needs separate metrics for return, constraint violations, uncertainty, robustness, and recoverability. Average return answers, "How well does it optimize the task?" Safety metrics answer, "What bad behavior did optimization create or fail to prevent?"

Evaluation should be broader than reward:

Evaluation target	What to test or report
Generalization	Held-out levels, seeds, layouts, visuals, objects, and dynamics ranges.
Robustness	Observation noise, occlusions, distractors, action noise, latency, stochastic resets, and degradation curves.
Uncertainty and risk	Policy entropy, ensemble variance, $CVaR_\alpha$ , worst-k percent return, and the tradeoff against mean return.
Safety	Constraint violation rate and severity, interventions, recovery time, near-misses, and worst-case return under stress.
Reproducibility	Seeds, compute, wall-clock time, gradient steps, training curves, ablations, and sensitivity to $\alpha$ , $\beta$ , or $\tau$ .

Safety evaluation

Reward is not the release gate

Choose a trained policy. The deployment gate checks average return, violation rate, and recoverability together.

avg return

84%

violations

recovery

76%

Candidate

This policy gives up some reward, but its safety slices stay inside the deployment gate.

A safe RL report should track unacceptable behavior explicitly, not hope it is encoded in average return.

A constrained RL problem can be written as maximizing return while keeping expected cost below a limit:

$\max_{\pi}\;\mathbb{E}_{\pi}[G_t]\quad\text{subject to}\quad\mathbb{E}_{\pi}[C_t]\le d$

Here $G_t$ is task return, $C_t$ is a safety cost such as collisions or unsafe refusals, and $d$ is the tolerated budget. In practice, the evaluation report should show both quantities instead of collapsing them into one score.

⚠

Stop Rules Matter

If average return rises while safety-slice performance stalls or falls, that is not a clean improvement. The right response is to pause, inspect failures, tighten constraints, or roll back.

Practical safety tools include action shielding, rule-based or learned filters, robust MDP training, domain randomization, and uncertainty penalties. The point is to make unsafe behavior hard to choose, then report whether that protection survives stress tests rather than only reporting the average task score.

Checkpoint

Which evaluation result is the strongest safety warning?

←PreviousGeneralization FailuresSafety, Generalization, and Exploration