repository·main·Indexed 26 days ago
https://github.com/kwai/douzeroA reinforcement learning framework designed to master the Chinese card game DouDizhu using self-play deep reinforcement learning. DouZero utilizes a Deep Monte Carlo (DMC) algorithm with action encoding and parallel actors to manage large state and action spaces. The framework includes tools for training models via train.py, generating evaluation data, and performing self-play evaluation using evaluate.py.
By default, models are saved every half hour in the douzero_checkpoints/douzero/ directory. You can use the provided shell script to find the path to the most recently saved checkpoint. The script will place the most recent model in the most_recent_model directory.
sh get_most_recent.sh douzero_checkpoints/douzero/To train DouZero, run the train.py script. By default, it trains on a single GPU.
To utilize multiple GPUs, use the following arguments:
--gpu_devices: Comma-separated list of visible GPU device indices.--num_actor_devices: Number of GPU devices used for simulation (self-play).--num_actors: Number of actor processes per simulation device.--training_device: The index of the GPU used for training.Example (4 GPUs: 3 for simulation with 15 actors each, 1 for training):
python3 train.py --gpu_devices 0,1,2,3 --num_actor_devices 3 --num_actors 15 --training_device 3To run training or simulation on the CPU (required for Windows actors):
--training_device cpu: Use CPU for training.--actor_device_cpu: Use CPU for actors.Example (Everything on CPU):
python3 train.py --actor_device_cpu --training_device cpuExample (Only actors on CPU):
python3 train.py --actor_device_cpupython3 train.py --gpu_devices 0,1,2,3 --num_actor_devices 3 --num_actors 15 --training_device 3Run the evaluate.py script to evaluate DouZero agents against different types of opponents (random, RLCard, or pre-trained models).
Key hyperparameters:
--landlord: The agent playing as the landlord. Options: random, rlcard, or a path to a pre-trained model checkpoint.--landlord_up: The agent playing as the landlord's upper neighbor. Options: random, rlcard, or a path to a pre-trained model checkpoint.--landlord_down: The agent playing as the landlord's lower neighbor. Options: random, rlcard, or a path to a pre-trained model checkpoint.--eval_data: The pickle file containing evaluation data.--num_workers: Number of processes to use for simulation.--gpu_device: The GPU device to use for simulation (defaults to CPU).You can install DouZero via pip. It requires Python 3.6+.
Standard Installation:
pip3 install douzeroInstallation for users in China (using Tsinghua mirror):
pip3 install douzero -i https://pypi.tuna.tsinghua.edu.cn/simpleDevelopment Installation (latest version):
pip3 install -e .Note on Hardware:
pip3 install douzeroEvaluation is performed via self-play. You can use pre-trained weights by placing them in the baselines/ directory.
Available Baselines:
random: Random agent (uniform selection).rlcard: Rule-based model from the RLCard project.sl: Deep learning pre-trained model based on human data (found in baselines/sl/).douzero_ADP: DouZero agent trained with ADP objective (baselines/douzero_ADP/).douzero_WP: DouZero agent trained with WP objective (baselines/douzero_WP/).Step 1: Generate evaluation data Run the following command to generate the necessary data for evaluation:
python3 generate_eval_data.pyImportant generate_eval_data.py arguments:
--output: Path to store the pickle data.--num_games: Number of games to generate (default: 10000).python3 generate_eval_data.pyEvaluation can be performed using a GPU (faster) or CPU. To evaluate, you must first generate evaluation data and then run the self-play evaluation script.
Prerequisites:
baselines/ directory.random (random agents), rlcard (rule-based agents), baselines/sl/ (SL agents), baselines/douzero_ADP/ (ADP objective), and baselines/douzero_WP/ (WP objective).Step 1: Generate evaluation data Run the data generation script to create pickled game data.
Step 2: Run self-play evaluation Run the evaluation script using the generated data and specifying the agents for the Landlord, LandlordUp, and LandlordDown positions.
# Step 1: Generate evaluation data
python3 generate_eval_data.py
# Step 2: Self-Play
python3 evaluate.py --landlord baselines/douzero_ADP/landlord.ckpt --landlord_up random --landlord_down randomoperation not supported error. This is caused by Windows not supporting multi-processing on CUDA tensors. Since the codebase is optimized for GPU and performs extensive CUDA tensor operations, this is a known limitation on Windows.operation not supported error while training on Windows using GPU actors, it is because Windows does not support multiprocessing on CUDA tensors. Since the code is optimized for GPUs and operates extensively on CUDA tensors, this is a known limitation on Windows systems.Arguments for the evaluate.py script:
| Flag | Description |
|---|---|
--landlord | Which agent will play as Landlord (can be random, rlcard, or a path to a pre-trained model) |
--landlord_up | Which agent will play as LandlordUp (the one playing before the Landlord; can be random, rlcard, or a path to a pre-trained model) |
--landlord_down | Which agent will play as LandlordDown (the one playing after the Landlord; can be random, rlcard, or a path to a pre-trained model) |
--eval_data | The pickle file containing the evaluation data |
--num_workers | Number of subprocesses to use |
--gpu_device | Which GPU to use (defaults to CPU) |
Configuration options for the train.py script:
| Argument | Description |
|---|---|
--xpid XPID | Experiment ID (default: douzero) |
--save_interval SAVE_INTERVAL | Interval to save models (in minutes) |
--objective {adp,wp} | Reward objective: adp (Average Difference Points) or wp (Winning Percentage). Default: adp |
--actor_device_cpu | Use CPU for simulation |
--gpu_devices GPU_DEVICES | GPU device names used for training |
--num_actor_devices NUM_ACTOR_DEVICES | Number of GPUs used for simulation (e.g., self-play) |
--num_actors NUM_ACTORS | Number of actor processes per device |
--training_device TRAINING_DEVICE | Device for model training (cpu or GPU ID). |
--load_model | Load an existing model |
--disable_checkpoint | Disable checkpoint saving |
--savedir SAVEDIR | Path to store experiment data |
--total_frames TOTAL_FRAMES | Total environment frames to train for |
--exp_epsilon EXP_EPSILON | Exploration probability |
--batch_size BATCH_SIZE | Training batch size |
--unroll_length UNROLL_LENGTH | Unroll length (time dimension) |
--num_buffers NUM_BUFFERS | Number of shared memory buffers |
--num_threads NUM_THREADS | Number of learner threads |
--max_grad_norm MAX_GRAD_NORM | Maximum gradient norm |
--learning_rate LEARNING_RATE | Learning rate |
--alpha ALPHA | RMSProp smoothing constant |
--momentum MOMENTUM | RMSProp momentum |
--epsilon EPSILON | RMSProp epsilon |
The following arguments are available for customizing the train.py execution:
| Argument | Description |
|---|---|
--xpid XPID | Experiment id (default: douzero) |
--save_interval SAVE_INTERVAL | Time interval (in minutes) at which to save the model |
--objective {adp,wp} | Use adp or wp as reward (default: adp) |
--actor_device_cpu | Use CPU as actor device |
--gpu_devices GPU_DEVICES | Which GPUs to be used for training |
--num_actor_devices NUM_ACTOR_DEVICES | The number of devices used for simulation |
--num_actors NUM_ACTORS | The number of actors for each simulation device |
--training_device TRAINING_DEVICE | The index of the GPU used for training models. cpu means using cpu |
--load_model | Load an existing model |
--disable_checkpoint | Disable saving checkpoint |
--savedir SAVEDIR | Root dir where experiment data will be saved |
--total_frames TOTAL_FRAMES | Total environment frames to train for |
--exp_epsilon EXP_EPSILON | The probability for exploration |
--batch_size BATCH_SIZE | Learner batch size |
--unroll_length UNROLL_LENGTH | The unroll length (time dimension) |
--num_buffers NUM_BUFFERS | Number of shared-memory buffers |
--num_threads NUM_THREADS | Number learner threads |
--max_grad_norm MAX_GRAD_NORM | Max norm of gradients |
--learning_rate LEARNING_RATE | Learning rate |
--alpha ALPHA | RMSProp smoothing constant |
--momentum MOMENTUM | RMSProp momentum |
--epsilon EPSILON | RMSProp epsilon |
generate_eval_data.py to create the pickled data required for the evaluation step.