repository·main·Indexed 20 days ago
https://github.com/uoe-agents/epymarlAn extended Multi-Agent Reinforcement Learning (MARL) framework built upon PyMARL. It supports modern environments including Gymnasium, PettingZoo, SMACv2, and SMAClite, as well as general-sum reward settings and advanced algorithms such as Pareto-AC, MAPPO, and QMIX. The framework includes tools for hyperparameter searching, Weights and Biases (W&B) logging, and result visualization.
Models are stored in the results/models directory.
Saving:
Set save_model = True in your config. The frequency is controlled by save_model_interval.
Loading:
Use checkpoint_path (pointing to the run directory) and load_step.
load_step is omitted, the last checkpoint is loaded.load_step is provided, the closest checkpoint to that timestep is loaded.evaluate=True.The plot_results.py script visualizes logged metrics. It supports window-smoothing, aggregating results across multiple runs, and filtering by algorithm/environment.
Use the --best_per_alg flag to only plot the best configuration for each algorithm instead of visualizing all configurations.
To set up the EPyMARL framework, follow these installation steps:
Core Dependencies: Clone the repository and install the base requirements:
pip install -r requirements.txtSupported Environments: To install a suite of supported environments (including Level Based Foraging, RWARE, PettingZoo, VMAS, Matrix games, SMAC, SMACv2, and SMAClite), run:
pip install -r env_requirements.txtPareto-AC (PAC) Dependencies: If you intend to use the Pareto-AC algorithm, you must install its specific dependencies:
pip install -r pac_requirements.txtNote: SMAC and SMACv2 require a StarCraft II installation with specific map files. For individual environment installation, refer to their respective repositories.
pip install -r requirements.txt
pip install -r env_requirements.txt
pip install -r pac_requirements.txtYou can run experiments for various benchmark environments using src/main.py. The command structure typically involves specifying a --config (the algorithm), an --env-config (the environment wrapper type), and env_args (environment-specific parameters).
Matrix Games (using gymma):
python src/main.py --config=qmix --env-config=gymma with env_args.time_limit=25 env_args.key="matrixgames:penalty-100-nostate-v0"Level Based Foraging (LBF) (using gymma):
python src/main.py --config=qmix --env-config=gymma with env_args.time_limit=50 env_args.key="lbforaging:Foraging-8x8-2p-2f-coop-v3"RWARE (using gymma):
python src/main.py --config=qmix --env-config=gymma with env_args.time_limit=500 env_args.key="rware:rware-tiny-2ag-v2"MPE (PettingZoo) (using gymma):
For MPE environments like simple adversary or simple tag, you can use pre-trained policies to make the task cooperative by setting env_args.pretrained_wrapper:
# For simple tag
python src/main.py --config=qmix --env-config=gymma with env_args.time_limit=25 env_args.key="pz-mpe-simple-tag-v3" env_args.pretrained_wrapper="PretrainedTag"SMAC (using sc2):
python src/main.py --config=qmix --env-config=sc2 with env_args.map_name="3s5z"# Example: Running QMIX on a specific SMAC map
python src/main.py --config=qmix --env-config=sc2 with env_args.map_name="3s5z"EPyMARL supports PettingZoo and VMAS via the gymma environment wrapper. Use the env_args.key argument to specify the environment ID.
# PettingZoo example
python src/main.py --config=qmix --env-config=gymma with env_args.time_limit=25 env_args.key="pz-mpe-simple-spread-v3"
# VMAS example
python src/main.py --config=qmix --env-config=gymma with env_args.time_limit=150 env_args.key="vmas-balance"EPyMARL provides wrappers for SMACv2 and SMAClite environments.
SMACv2: Use the sc2v2 environment config. You can specify scenarios using env_args.map_name. Prepared configs are located in src/config/envs/smacv2_configs.
SMAClite: Use the smaclite environment config. You can specify the map via env_args.map_name. To use the optimized C++ RVO2 library instead of the default numpy implementation, set env_args.use_cpp_rvo2=True (requires external installation of the RVO2 library).
# SMACv2 example
python src/main.py --config=qmix --env-config=sc2v2 with env_args.map_name="protoss_5_vs_5"
# SMAClite example
python src/main.py --config=qmix --env-config=smaclite with env_args.time_limit=150 env_args.map_name="MMM"
# SMAClite with optimized C++ RVO2
python src/main.py --config=qmix --env-config=smaclite with env_args.time_limit=150 env_args.map_name="MMM" env_args.use_cpp_rvo2=TrueThe search.py script executes hyperparameter searches based on a configuration file.
locally argument to run the search on your machine.single <index> argument to run a specific hyperparameter combination as a single process, useful for batch scripts in cluster environments.# Local hyperparameter search
python search.py run --config=search.config.example.yaml --seeds 5 locally
# Cluster execution (running the 1st configuration in a batch)
python search.py run --config=search.config.example.yaml --seeds 5 single 1To use a custom Gymnasium environment, register it using the standard gymnasium.register API. Then, run it using the gymma environment config, passing the registration ID (including the package prefix if applicable) to env_args.key.
Note on Rewards:
common_reward=True).common_reward=False).# Registration template
from gymnasium import register
register(
id="my-environment-v1", # Environment ID
entry_point="myenv.environment:MyEnvironment", # Entry point for the class
kwargs={ ... }, # Arguments for __init__
)# Running the custom environment
python src/main.py --config=qmix --env-config=gymma with env_args.time_limit=50 env_args.key="myenv:my-environment-v1"EPyMARL supports logging experiment data to Weights and Biases (W&B).
pip install wandb.pip install wandbEPyMARL supports training in environments with individual rewards for all agents (general-sum reward settings).
IA2C, IPPO, MAA2C, MAPPO, IQL, and PAC. Algorithms like COMA, VDN, QMIX, and QTRAN only support common-reward environments.common_reward=False in your execution command.common_reward=True is used in an individual-reward environment, EPyMARL defaults to summing all rewards. You can change this to the mean operation by setting reward_scalarisation="mean".Example running MAPPO in a LBF task with individual rewards:
python src/main.py --config=mappo --env-config=gymma with env_args.time_limit=50 env_args.key="lbforaging:Foraging-8x8-2p-3f-v3" common_reward=FalseTo log results to Weights and Biases, ensure the wandb library is installed and authenticated. Add the following parameters to your YAML configuration:
use_wandb: True: Enables W&B logging.wandb_team: <name>: Specifies the W&B team.wandb_project: <name>: Specifies the W&B project.wandb_mode: "online": Forces online logging (default is "offline").Model Logging: To upload trained models to the W&B dashboard, set:
save_model: Truewandb_save_model: True (requires use_wandb: True)save_model_interval: <int>use_wandb: True
wandb_team: "my-team"
wandb_project: "my-project"
wandb_mode: "online"
# To also save and upload models
save_model: True
wandb_save_model: True
save_model_interval: 50000EPyMARL uses YAML files for configuration, located in src/config:
src/config/default.yaml: Defines default values for experiment info (e.g., t_max) and hyperparameters.src/config/algs/: Contains algorithm-specific configurations (passed via --config).src/config/envs/: Contains environment-specific configurations (passed via --env-config).