AlphaZero as a Motivating Example
A useful case study asks why machines reached grandmaster chess in the 1990s and took about two more decades to defeat professional Go players. The answer includes the structure of Go: vastly more board configurations, many more legal moves per turn, longer games, no fixed pieces, and sparser reward.
AlphaZero matters because it shows a general-purpose RL recipe working across games. It starts with randomly initialized neural network weights, plays games against itself, uses Monte Carlo Tree Search to improve action probabilities, records training examples, trains the network, and repeats.
AlphaZero
Follow the AlphaZero loop
Step through the clean version of the process: network policy, MCTS-improved policy, self-play data, training, repeat.
Network policy
Move at C3: 0.35
Move at F5: 0.20
Move at H7: 0.10
The policy is a probability distribution over all legal actions in a given state.
Instead of choosing a single move, the network assigns a probability to each possible move.
Learning objective
The network outputs a probability distribution over moves.
Read the three probabilities. Which move would the current network sample most often?
1 / 6
For each state in a self-play game, AlphaZero stores three things:
Here is the board state, is the search-improved probability distribution over actions, and is the final outcome of the game. A neural network is trained to predict both a policy and a value from the board state.
This is a beautiful example because the environment is closed and evaluable. The rules are known. The outcome is clear. The same structure becomes much harder when states are messy, rewards are ambiguous, or harmful exploration is unacceptable.
Why Self-Play Works Here
Self-play creates a curriculum automatically. As the current policy improves, it generates stronger opponents for itself. The data distribution rises with the policy.
In AlphaZero training, what does the stored outcome represent?