DOCSEARCHbyAlgolia
Submit Docs

MARLlib: Multi-agent Reinforcement Learning Library

repository·master·Indexed 23 days ago

https://github.com/replicable-marl/marllib

A Multi-agent Reinforcement Learning library built on Ray and RLlib. MARLlib provides a unified platform for developing, training, and testing MARL algorithms across cooperative, competitive, and mixed-mode environments. It includes 18 pre-built algorithms, a Gym-like interface, and support for various environments such as SMAC, MPE, and Hanabi. The library features a streamlined workflow for environment preparation, algorithm initialization, model building, and training via a .fit() method.

Tokens
28.7K
Snippets
47
Records
132
Agent score
78%

What's inside MARLlib

  1. Overview of available MARL benchmarks

    master

    MARLlib aggregates several existing Multi-Agent Reinforcement Learning (MARL) benchmarks and libraries. These include:

    • PyMARL: The original library focused on SMAC environments for cooperative learning.
    • PyMARL2: An extension focusing on credit assignment mechanisms and fine-tuned QMIX for SMAC.
    • EPyMARL: An extension aimed at unifying cooperative MARL algorithms (independent learning, value decomposition, and centralized critic) across environments like LBF, RWARE, and MPE.
    • MARL-Algorithms: A library covering credit assignment, communication-based, graph-based, and multi-task curriculum learning, primarily tested on SMAC.
    • MAPPO benchmark: The official codebase for MAPPO, focusing on cooperative MARL across four environments.
    • MAlib: A library for population-based MARL that combines game theory with MARL algorithms for meta-game tasks.

  2. What is MARLlib?

    master

    MARLlib (Multi-Agent Reinforcement Learning Library) is a comprehensive library for building, training, and evaluating Multi-Agent Reinforcement Learning algorithms. It is built upon Ray and its toolkit RLlib.

    Key capabilities include:

    • Unified Pipelines: Uses agent-level distributed dataflow to unify diverse algorithm pipelines across different tasks and environments.
    • Task Mode Support: Supports cooperative, collaborative, competitive, and mixed task modes.
    • Gym-like Interface: Provides a new interface following the Gym structure for easier multi-agent environment interaction.
    • Customizable Strategies: Offers flexible parameter-sharing and grouping options for policies.
    • Model Zoo: Provides access to pre-built model architectures and 18 pre-built algorithms.

  3. Key Features of MARLlib

    master

    MARLlib is designed to unify diverse algorithm pipelines with agent-level distributed dataflow. Its primary features include:

    • Unified Pipeline: Supports developing, testing, and evaluating MARL algorithms across different tasks and environments.
    • Task Mode Support: Supports cooperative, collaborative, competitive, and mixed task modes.
    • Gym-like Interface: Provides a new interface following the Gym structure for easier multi-agent environment interaction.
    • Flexible Parameter Sharing: Offers customizable parameter-sharing strategies (share, group, separate, or custom).
    • Model Customization: Allows choosing from a model zoo or building custom architectures.
    • Pre-built Algorithms: Includes 18 pre-built algorithms with an intuitive API.

  4. Explore MARL algorithms by environment (MPE and SMAC)

    master

    MARLlib provides a curated list of Multi-Agent Reinforcement Learning (MARL) algorithms categorized by the environments they are designed for or commonly evaluated on. This allows developers to select algorithms based on the specific complexity and characteristics of their target environment.

    Supported Environment Categories

    • MPE (Multi-Agent Particle Environment): Focuses on diverse scenarios including mixed cooperative-competitive tasks, communication, and robust learning.
    • SMAC (StarCraft Multi-Agent Challenge): Focuses on cooperative multi-agent learning, value decomposition, and complex coordination tasks.

    Algorithm Annotation Tags

    Algorithms are annotated with specific sub-topic tags to help you find relevant research:

    • [B] Basic
    • [S] Information Sharing
    • [RG] Behavior/Role Grouping
    • [I] Imitation
    • [G] Graph
    • [E] Exploration
    • [R] Robust
    • [P] Reward Shaping
    • [F] Offline
    • [T] Tree Search
    • [MT] Multi-task

  6. What is VDA2C (Value Decomposition Advanced Actor-Critic)?

    master

    VDA2C is an extension of the IA2C algorithm designed specifically for cooperative and collaborative multi-agent tasks. It focuses on credit assignment by using a value function ($V$ function) rather than a $Q$ function.

    Key Characteristics:

    • Task Modes: Cooperative, Collaborative
    • Action Spaces: Supports both discrete and continuous control
    • Learning Type: On-policy, stochastic, value decomposition
    • Architecture: Consists of three modules: policy, critic, and mixer.

    Workflow: During sampling, agents exchange observations and predicted critic values. The training follows the standard A2C pipeline, but uses a mixed critic value to calculate the Generalized Advantage Estimation (GAE) and perform the critic learning procedure.

  7. What is IA2C (Independent Advanced Actor-Critic)?

    master

    IA2C is a multi-agent extension of the standard Advanced Actor-Critic (A2C) algorithm. In IA2C, each agent operates as an independent A2C-based sampler and learner. It is a versatile baseline suitable for various multi-agent reinforcement learning (MARL) tasks because it does not strictly require information sharing to function, though sharing is supported.

    Key Characteristics:

    • Agent Architecture: Consists of two modules: policy and critic.
    • Task Modes: Supports cooperative, collaborative, competitive, and mixed modes.
    • Action Spaces: Supports both discrete and continuous action spaces.
    • Learning Type: It is an on-policy, stochastic, independent learning algorithm.
    • Information Sharing: While agents can act independently, you can choose to share knowledge (such as experience replay buffers or model parameters), predicted data (Q/critic values), or real/sampled data (observations, actions).

  8. What is the Trust Region Policy Optimization (TRPO) family?

    master

    The TRPO family consists of policy-gradient based reinforcement learning algorithms that ensure policy updates remain within a defined 'trust region' to maintain stability. This family includes:

    1. TRPO: The standard single-agent algorithm.
    2. ITRPO (Independent TRPO): A multi-agent extension where each agent follows a standard TRPO pipeline independently. It does not require information sharing during training, though knowledge sharing (sharing models/buffers) is possible.
    3. MATRPO (Multi-agent TRPO): An extension of ITRPO that uses a centralized critic. During sampling, agents share observations and predicted actions. The centralized value function uses the global state and the actions of all other agents to calculate Generalized Advantage Estimation (GAE), allowing agents to account for teammates' behaviors.

    All algorithms in this family are on-policy and support both discrete and continuous action spaces across cooperative, collaborative, competitive, and mixed task modes.

  9. Overview of MARLlib Configuration

    master

    To customize the MARL learning pipeline in MARLlib, you must manage four distinct configuration types:

    1. Scenario: Defines environment and task settings.
    2. Algorithm: Used to finetune algorithm-specific hyperparameters.
    3. Model: Used to customize the agent's model architecture (e.g., encoders, mixers).
    4. Ray/RLlib: Controls training infrastructure settings like sampling speed, GPU acceleration, and distributed running modes.

  10. Understand the Joint Q Learning Family

    master

    The Joint Q Learning family of algorithms in MARLlib includes methods that extend single-agent Q-learning to multi-agent settings. This family covers:

    1. Deep (Recurrent) Q Learning (DQN/DRQN): The foundation of Q-learning using neural networks to approximate Q-values. DRQN extends this to Partially Observable Markov Decision Processes (POMDPs) by using Recurrent Neural Networks (like GRU or LSTM) to encode history.
    2. Independent Q Learning (IQL): A multi-agent extension where each agent treats the environment as if it were alone, learning its own Q-function independently. It is applicable to cooperative, collaborative, competitive, and mixed task modes.
    3. Value Decomposition Networks (VDN): (Details in subsequent segments) Uses a mixing network to decompose the joint Q-value.
    4. QMIX: (Details in subsequent segments) Uses monotonic factorization for mixing Q-values.

  11. Understand the Deep Reinforcement Learning (DRL) learning cycle

    master

    Deep Reinforcement Learning (DRL) combines Reinforcement Learning (RL) with Deep Learning (DL) to enable agents to make decisions from unstructured input data without manual state space engineering. The learning process follows a cycle involving data collection, batch formation, and policy optimization.

    There are two primary learning paradigms:

    • On-policy: The agent learns from data collected by the current policy.
    • Off-policy: The agent learns from data collected by a different policy (often stored in a replay buffer).

    Key stages in the cycle include:

    1. Data collection: The agent sends an action to the environment; the environment returns observations, states, rewards, etc.
    2. Form a batch: Data from collection is used to form batches for stochastic gradient descent (SGD).
    3. Replay buffer: Data is sent to a replay buffer for future optimization use (primarily in off-policy learning).
    4. Sample a batch: A batch is sampled from the replay buffer following specific rules.
    5. Policy optimization: The sampled data batch is used to optimize the policy.

  12. Understand Knowledge Sharing Levels in MARL

    master

    Knowledge sharing is used to increase sample efficiency and adaptability. It occurs at three distinct levels:

    • Agent Level: Sharing model parameters or replay buffers between agents. While this improves learning speed, it can reduce the diversity of individual policies, which may be detrimental in competitive scenarios.
    • Scenario Level: Developing a general policy for multiple scenarios within the same task domain (e.g., different maps or unit types in SMAC). This often utilizes transformer-based architectures or meta-learning.
    • Task Level: The most advanced level, where agents learn to generalize principles across entirely different task domains (e.g., applying teamwork concepts learned in one domain to a completely new one).