Introduction to Modern Reinforcement Learning

Decision Transformers approach offline RL from a different angle: if we know what return we want, can we predict the actions that tend to get there using a supervised sequence model? Instead of learning a Q-function and maximizing it, the model conditions on a desired return, the past states, and the past actions, then predicts the next action.

This reframes RL as conditional sequence modeling. The dataset becomes a supervised training set: when the remaining target return was high and the agent saw this state history, this action came next. The model learns patterns in successful trajectories without explicit TD bootstrapping, value functions, or Q-maximization over unseen actions.

Decision Transformer diagram showing return, state, and action tokens processed as a sequence model. — Decision Transformer frames offline reinforcement learning as conditional sequence modeling over returns, states, and actions. [Source]

A Decision Transformer trains on token sequences such as

$(R_t,s_t,a_t),(R_{t+1},s_{t+1},a_{t+1}),\ldots$

The model learns

$a_t=f_\theta(R_t,s_t,a_{<t},s_{<t})$

or, as a likelihood objective,

$\mathcal{L}_{\text{DT}}=\mathbb{E}\left[-\log\pi_\theta(a_t\mid R_t,s_t,\text{history})\right]$

The architecture is usually a causal transformer: masked self-attention only, no encoder half, and each token can attend to past tokens but not future ones. In other words, the model tokenizes decisions and predicts the next action autoregressively.

Decision Transformer architecture figure from Lee et al. showing trajectory tokens and causal prediction. — Decision Transformer-style models arrange trajectory information as tokens and train with causal prediction. [Source]

At inference time, the loop is simple:

Step 1

Set target return

Choose the return you want the trajectory to achieve, such as $R_0$ .

->

Step 2

Predict action

Use return, current state, and history to predict the next action.

->

Step 3

Environment responds

The action produces the next state and reward.

->

Step 4

Update and repeat

Adjust the remaining target return, append the new context, and continue.

Loop: new state + reward become part of the next prediction context

Decision Transformer inference repeatedly conditions on the desired return and accumulated history.

This can feel like "RL without RL": the training objective is supervised action prediction, but the conditioning variable lets the user ask for higher-return behavior when the dataset supports it.

Decision Transformer inference figure from Lee et al. illustrating return-conditioned action generation. — An overview of the training and evaluation setup. Expert-level game-play is observed in the interactive setting after offline learning from trajectories ranging from beginner to expert. [Source]

The limitation is still coverage. If you ask for a return higher than anything in the dataset, or for behavior in states the dataset barely covers, the transformer is also extrapolating. Decision Transformers avoid one failure mode of Q-learning, but they do not make unsupported deployment safe.

ℹ

When Decision Transformers Shine

They are most natural when you have large trajectory datasets with varied returns: game play traces from experts and non-experts, robot logs across many tasks, navigation data with different path qualities, or simulated rollouts covering many styles of behavior.

Method comparison

Choose the offline RL tool

Select a method to compare what signal it uses, how it handles unsupported actions, and when it is most natural.

Selected method

CQL

CQL still learns Q-values, but it pushes down unsupported optimism so the policy does not chase actions the dataset cannot justify.

Uses rewards?

Yes

Handles OOD actions?

Strongly.

Strength

Conservative Q-values.

When to use

Mixed datasets where Q-learning may overrate unsupported actions.

All four methods are responses to the same constraint: the dataset is fixed, so unsupported action choices need special care.