repository·main·Indexed 20 days ago
https://github.com/facebookresearch/benchmarlA Multi-Agent Reinforcement Learning (MARL) training and benchmarking library built on TorchRL. It provides a standardized framework for comparing MARL algorithms, environments, and models with a focus on reproducibility and statistical rigor. The library supports custom algorithm and model implementation, ensemble configurations via EnsembleAlgorithmConfig and EnsembleModelConfig, and integration with Hydra for configuration and Weights & Biases for hyperparameter sweeps.
BenchMARL is a Multi-Agent Reinforcement Learning (MARL) training library designed for reproducibility and benchmarking. It provides a standardized interface for integrating new algorithms and environments, ensuring fair comparisons.
Key features include:
BenchMARL is a Multi-Agent Reinforcement Learning (MARL) training library designed for reproducibility and benchmarking. It provides a standardized interface to integrate new algorithms and environments, enabling fair comparisons between different MARL solutions.
Key Technical Stack:
TorchRL and PyTorch for high performance and state-of-the-art implementations.Hydra for flexible and modular configuration.marl-eval for standardized and statistically strong evaluations.Core Design Tenets:
The benchmarl.models module provides the core abstractions for defining neural network architectures used in BenchMARL. It is organized into two main categories:
Model, ModelConfig, SequenceModel, and SequenceModelConfig.Model interface.BenchMARL is designed to be extensible. You can implement your own solutions by inheriting from the provided base classes and implementing their abstract methods. These base classes leverage TorchRL objects.
benchmarl/algorithms/common.pybenchmarl/environments/common.pybenchmarl/models/common.pyBenchMARL is a unified training library that interconnects components from TorchRL to enable fair and reproducible Multi-Agent Reinforcement Learning (MARL) benchmarking. The library uses a hierarchical structure of components:
Experiment fixes an Algorithm, a Task (environment), and a Model, along with a specific seed and hyperparameters.To ensure reproducibility, BenchMARL provides default configurations for each component. While experiment hyperparameters can be modified, task configurations should generally remain unchanged.
BenchMARL uses a dual approach for configuration: it can be managed directly within a Python script or via hydra.
All components (Experiments, Algorithms, Tasks, and Models) have corresponding YAML configuration files located in the benchmarl/conf directory. These YAML files are loaded into Python dataclasses which serve as schemas. This provides both separation of configuration from code and strong typing for parameter name and type validation.
If you need to load and validate a configuration YAML file manually in a script without using Hydra, you can use the .get_from_yaml() method available on the component's configuration dataclass.
BenchMARL is a unified training library for Multi-Agent Reinforcement Learning (MARL) that uses TorchRL as a backend. It is built around several key abstractions:
benchmarl/conf/experiment/base_experiment.yaml).Note on Agent Grouping: BenchMARL uses the TorchRL MARL API for grouping agents. In competitive environments, agents in different teams belong to different groups, each having its own loss, models, and buffers. Parameter sharing applies within these groups.
BenchMARL is designed to be extensible, allowing you to benchmark new solutions by implementing standard interfaces. To introduce a custom solution, you must implement the abstract methods provided by the following base classes:
benchmarl.algorithms.Algorithm: For implementing new learning algorithms.benchmarl.environments.Task: For defining new benchmark environments or tasks.benchmarl.models.Model: For implementing custom neural network architectures.These base classes utilize objects from the TorchRL library. Detailed implementation examples for each can be found in the repository's examples directory.
# Example links for implementation patterns:
# Custom Algorithm: https://github.com/facebookresearch/BenchMARL/blob/main/examples/extending/algorithm
# Custom Task: https://github.com/facebookresearch/BenchMARL/blob/main/examples/extending/task
# Custom Model: https://github.com/facebookresearch/BenchMARL/blob/main/examples/extending/modelBenchMARL allows assigning different algorithms or models to different agent groups using EnsembleAlgorithmConfig or EnsembleModelConfig.
Ensemble Algorithms:
Ensemble Models:
from benchmarl.algorithms import EnsembleAlgorithmConfig, IsacConfig, MaddpgConfig
from benchmarl.models import EnsembleModelConfig, GnnConfig, MlpConfig
# Ensemble Algorithm Example
algorithm_config = EnsembleAlgorithmConfig(
{"agent": MaddpgConfig.get_from_yaml(), "adversary": IsacConfig.get_from_yaml()}
)
# Ensemble Model Example
model_config = EnsembleModelConfig(
{"agent": MlpConfig.get_from_yaml(), "adversary": GnnConfig.get_from_yaml()}
)You can generate various benchmark visualizations including aggregate scores, sample efficiency curves, performance profiles, and probability of improvement plots by running the plot_benchmark.py script. This script utilizes the marl-eval library to produce these plots.
To use this, ensure you have marl-eval installed and then execute the plotting script against your benchmark data.
python plot_benchmark.pyIf you plan to use Graph Neural Network (GNN) models within BenchMARL, you must install torch_geometric:
pip install torch_geometricpip install torch_geometricBenchMARL requires torchrl. You can install it from PyPi:
pip install torchrlFor nightly versions or more advanced installation options, refer to the official TorchRL installation guide.
pip install torchrl