Learning from Fixed Datasets

Offline reinforcement learning, sometimes called batch RL, learns a policy from a fixed dataset of past experience.

This is useful because online trial and error can be unacceptable. You do not want a medical dosing agent experimenting on real patients, a financial-services agent losing real money to discover risk, or a conservation agent testing harmful interventions on animals. Offline RL asks whether we can reuse historical data to improve decisions without collecting new risky experience first.

Motivating example for why learning from logged experience matters when live trial and error is costly, unsafe, or impossible.

The fixed dataset is usually a collection of trajectories:

$D=\{(s_t,a_t,r_t,s_{t+1})\}_{t=1}^{N}$

Each tuple records a state, the action taken by the behavior policy, the reward or outcome observed, and the next state. The behavior policy $\beta(a\mid s)$ is the process that generated the data: clinicians, human drivers, old software, a scripted controller, a previous RL policy, or a mixture of all of them.

Equation walkthrough

Read an offline RL dataset

Click each part of the notation to unpack what the fixed dataset contains.

=(,,,)

Transition tuple

one step of experience

A transition tuple is the basic unit of offline RL data: state now, action taken, reward observed, and next state reached.

Terms

Offline RL trains from D, a fixed collection of transition tuples; no new trial-and-error data arrives during training.

Fixed dataset

How big is the island of evidence?

Choose the historical dataset. Offline RL can only learn from what is already in the logs, so action support becomes a safety boundary.

Coverage

82%

Behavior quality

61%

Safe reuse

65%

Follow log habit78%

Known improvement69%

Rare maneuver51%

Unseen shortcut38%

Offline diagnosis

Offline RL can search for better in-support choices, but it must avoid trusting unsupported actions.

In offline RL, the data is not just examples. It is the region where value estimates are grounded.

ℹ

Concrete Example

Suppose a hospital has logged sepsis treatment decisions. The data includes vitals, lab values, medications, fluids, outcomes, and next states. Offline RL might look for policies that improve survival or reduce complications. But if almost no patient with a certain profile received a particular drug, the dataset provides weak evidence about that counterfactual action.

⚠

Dataset Support Is a Safety Boundary

If the learned policy chooses actions rarely or never seen in the dataset, it is asking the value function to guess. In notation, an offline policy can become dangerous when it selects an action $a_\pi$ that is not in the support of the logged data: $a_\pi \notin \operatorname{support}(D)$ . In high-stakes domains, "the model guessed this action has high value" is not enough evidence for deployment.

Checkpoint

Why is dataset support central in offline RL?

Next→Distributional MismatchOffline RL