Gato as RL-Generated Data

Gato extends the arc from RL agents to RL-generated data. Gato uses RL in a different way from AlphaZero: some of its datasets come from RL environments, including Atari rollouts, robot control traces, and other trajectories produced by agents trained with reinforcement learning. The model then imitates mappings from observations to actions across many domains.

AlphaZero shows RL as self-play and search. Gato shows trajectories from RL systems becoming data for broader generalist systems. RLHF, on the next page, shows RL as preference-shaped policy optimization.