AlphaZero.jl

AlphaZero.jl is a generic, simple, and fast Julia implementation of DeepMind's AlphaZero algorithm. It is designed to be accessible for researchers and students while maintaining high performance. Key features include:

• Simplicity: The core algorithm consists of approximately 2,000 lines of pure Julia code.
• Extensibility: Uses generic interfaces to allow easy integration of new [games](@ref game_interface) and new [learning frameworks](@ref network_interface).
• Performance: It is 10x to 100x faster than many Python alternatives, enabling the solving of non-trivial games on a standard desktop with a GPU.
• Scalability: The same agent can be trained on a single computer or a cluster of machines without code modifications.

To assess the strength of the trained agent, use the following report types:

• Report.Evaluation: Results from specific evaluation runs.
• Report.Benchmark: Performance metrics against standardized benchmarks.

The Learning Phase tracks the optimization of the neural network. Key reporting components include:

• Report.Learning: High-level learning progress.
• Report.Checkpoint: Information regarding saved model weights and checkpoints.
• Report.LearningStatus: The current state of the learning process.
• Report.Loss: Detailed loss metrics (e.g., policy loss, value loss) used to monitor convergence.

Training reports in alphazero.jl are organized into specific phases that track the progress of the AlphaZero algorithm. The reporting system is structured around the following lifecycle stages:

• Self-Play Phase: Reports on the generation of games through self-play.
• Memory Analysis Phase: Provides insights into the stored experience, including memory usage and sample distributions.
• Learning Phase: Tracks the neural network training process, including loss metrics, checkpoints, and learning status.
• Evaluations and Benchmarks: Contains data regarding agent performance against specific benchmarks or evaluation opponents.

AlphaZero.jl distinguishes between a game specification and a game environment to separate static game rules from dynamic game states:

• Game Specification (AbstractGameSpec): Holds all static information that does not change regardless of the current state (e.g., board dimensions, number of players, or the set of all possible actions).
• Game Environment (AbstractGameEnv): Holds the mutable state of a specific game instance (e.g., current piece positions, whose turn it is, or the current score).

When implementing a new game, you must provide both an AbstractGameSpec and an AbstractGameEnv that work in tandem.

AlphaZero.jl achieves its speed through two primary mechanisms:

1. Julia's inherent speed: Unlike Python implementations where tree search can become a significant bottleneck, Julia's performance allows the search component to run efficiently.
2. Asynchronous simulation mechanism: The implementation uses an asynchronous mechanism that batches requests to the neural network across multiple simulation threads. This maximizes GPU utilization by ensuring the hardware is constantly processing batches of data rather than waiting on individual simulation steps.

AlphaZero combines search and learning using two primary components:

1. Two-headed Neural Network: Acts as the learned heuristic. Given a board position, it outputs:
   • An estimate of the probability for each player to win (value head).
   • A probability distribution over available moves (policy head).
2. Monte-Carlo Tree Search (MCTS): Acts as the search component. It uses the neural network's heuristics to manage uncertainty and provides a probability distribution over moves rather than a single choice. It serves as a policy improvement operator by providing a better move distribution than the raw network output.

During training, the network is updated iteratively through self-play so that its predictions match the actual game outcomes and its policy matches the MCTS output.

An experiment in AlphaZero.jl is a bundle of all information required to spawn a training session. It is defined by the following fields:

Experiments are typically defined in game-specific files (e.g., games/connect-four/params.jl).