Conservative Objectives
Conservative offline RL methods respond to distributional mismatch by limiting how much the learned policy can trust unsupported actions. Conservatism is not pessimism for its own sake. It is a way to say: high value should require evidence.
Different algorithms enforce this idea differently. Behavioral cloning stays exactly on the dataset support. Conservative Q-Learning, or CQL, pessimistically penalizes out-of-distribution actions. Implicit Q-Learning, or IQL, avoids evaluating arbitrary out-of-distribution actions by learning from good in-support actions through advantage filtering. BCQ, BRAC, and AWAC are other examples of the same family: constrain the learned policy to the dataset manifold, or make leaving it expensive.
Offline RL team selection
Build a hackathon team from fixed evidence
You want the best team of three people. Each round changes how far you are allowed to move away from the people you already know.
Behavioral cloning
Choose only people you know. This imitates the dataset exactly.
Selected team
Select up to three people from the image.
Round feedback
Select 3 more candidates to complete the team.
Dataset in this analogy
Mike, Mrinal, Sam, and Tiffany are known candidates. Their traits define the support that BC, CQL, and IQL reason from.
Behavior cloning trains a policy to imitate dataset actions.
Equation walkthrough
Behavioral cloning imitates the dataset
Click each term to see how BC turns logged actions into a supervised learning objective.
Learned policy
the policy being trained
This is the policy with parameters theta. Behavioral cloning trains it to choose actions like the ones in the dataset.
Terms
It is a good baseline when the dataset is expert-like. If the historical behavior is near optimal, imitation can be strong, stable, and easy to evaluate. If the dataset contains mediocre behavior, cloning copies the mediocrity.
CQL tries to learn values while penalizing actions outside support. A simplified way to remember the idea is: lower Q-values for actions the dataset does not justify, and keep dataset actions comparatively high. This helps prevent the policy from exploiting optimistic Q-errors.
Equation walkthrough
CQL adds a conservative penalty
Click each term to see how CQL combines TD learning with pressure against unsupported high Q-values.
CQL objective
the full training loss
This is the loss CQL optimizes. It combines ordinary value learning with an extra conservative pressure against unsupported high Q-values.
Terms
Equation walkthrough
IQL improves by filtering logged actions
Click each equation or term to see how IQL avoids maximizing Q over out-of-distribution actions.
Advantage
logged action minus state baseline
Advantage asks whether the logged action was better or worse than the baseline value for that state.
Terms
Choosing a Method
Use behavior cloning when expert demonstrations are already strong. Use CQL-style conservatism when the dataset is mixed and Q-learning might overrate unsupported actions. Use IQL-style filtering when the dataset contains good trajectories mixed with worse ones and you want a stable way to improve within support. In practice, every offline RL algorithm is an answer to the same problem: improve on the logs without letting the policy drift into unsupported action space.
The Cost of Conservatism
A conservative policy may reject a truly good action if the dataset barely contains it. That is the tradeoff: avoid dangerous extrapolation, but accept that some improvements require online data, simulation, expert review, or a better dataset.
When is a conservative offline RL policy preferable to an apparently high-value policy?