DOCSEARCHbyAlgolia
Submit Docs

XuanCe: A Comprehensive and Unified Deep Reinforcement Learning Library

repository·master·Indexed 21 days ago

https://github.com/agi-brain/xuance

XuanCe is a modular, multi-backend Deep Reinforcement Learning library supporting PyTorch, TensorFlow2, and MindSpore. It provides high-quality implementations of DRL, Model-Based RL (MBRL), Multi-Agent RL (MARL), and Contrastive RL (CRL) algorithms. The framework supports distributed training across CPU, GPU, Linux, Windows, and MacOS, with built-in tools for hyperparameter tuning and visualization via Tensorboard and WandB.

Tokens
117.3K
Snippets
239
Records
484
Agent score
77%

What's inside xuance

  1. Overview of Communication Neural Net (CommNet)

    master

    CommNet is a Multi-Agent Reinforcement Learning (MARL) algorithm designed for cooperative tasks where agents have only local observations. It enables agents to autonomously learn continuous communication protocols through a shared channel.

    Key characteristics:

    • Centralized Training With Decentralized Execution (CTDE): Uses a central controller during training, which is removed during execution.
    • On-policy: The evaluation policy is the same as the target policy.
    • Model-free: Does not require an environment dynamics model.
    • Action Spaces: Supports both Discrete and Continuous action spaces.
    • Permutation Invariant: The order of agents does not affect the outcome, making it suitable for dynamic environments.

  2. Overview of XuanCe DRL Library

    master

    XuanCe is an open-source ensemble of Deep Reinforcement Learning (DRL) algorithm implementations. It is designed to be a highly modular, user-friendly, and high-performance 'DRL zoo' for researchers and developers.

    Key features include:

    • Multi-Backend Support: Compatible with PyTorch (>=1.13.0), TensorFlow (>=2.6.0), and MindSpore (>=1.10.1).
    • Versatile Task Support: Handles Single-Agent DRL, Multi-Agent Reinforcement Learning (MARL), Model-Based RL (MBRL), Contrastive RL (CRL), and Offline RL.
    • High Performance: Leverages vectorized environments and supports distributed multi-GPU training.
    • Advanced Capabilities: Supports automatic hyperparameter tuning and visualization via TensorBoard and Weights & Biases (wandb).
    • Cross-Platform: Runs on Linux, Windows, and macOS across CPU and GPU.

  3. Overview of MPE Tasks

    master

    The Multi-Agent Particle Environment (MPE) provides a suite of cooperative, competitive, and mixed-scenario tasks for multi-agent reinforcement learning (MARL).

    Available task types include:

    • Cooperative Navigation: simple_v3, simple_spread_v3
    • Cooperative-Push: simple_push_v3
    • Adversarial Navigation: simple_adversary_v3
    • Predator-Prey: simple_tag_v3
    • Communication Tasks: simple_reference_v3, simple_crypto_v3, simple_speaker_listener, simple_world_comm

  4. Overview of Weighted Q-Mixing Networks (WQMIX)

    master

    Weighted Q-Mixing Networks (WQMIX) is a value-based Multi-Agent Reinforcement Learning (MARL) algorithm designed to improve upon the QMIX algorithm. It addresses the limitations of QMIX's monotonicity constraint, which can prevent the model from accurately representing complex interdependencies between agents' actions.

    Key Features of WQMIX:

    • Centralized Training With Decentralized Execution (CTDE): Uses a central controller during training, which is discarded during execution.
    • Off-policy: The evaluation policy differs from the target policy.
    • Model-free: Does not require an environment dynamics model.
    • Discrete Action Space: Specifically designed for environments with discrete actions.
    • Decentralized Execution: Agents do not communicate with each other during execution.

  5. Overview of Q-Transformation (QTRAN)

    master

    QTRAN is a value decomposition method for Multi-Agent Reinforcement Learning (MARL) designed to overcome the structural limitations of classical algorithms like VDN and QMIX. While VDN uses additive decomposition and QMIX relies on a monotonicity constraint, QTRAN introduces a transformed joint action-value function. This allows it to represent a broader range of cooperative strategies, making it particularly effective for non-monotonic tasks (e.g., cooperative predation) where QMIX and VDN fail.

    Key Characteristics:

    • Paradigm: Centralized Training with Decentralized Execution (CTDE).
    • Policy Type: Off-policy and Model-free.
    • Action Space: Supports Discrete Action spaces (does not support Continuous Action spaces).
    • Communication: Neither fully decentralized nor fully centralized; it uses a central controller during training which is then abandoned during execution.

  6. Overview of the XuanCe common module

    master

    The common module in XuanCe provides a collection of reusable tools designed to be independent of the specific Deep Learning (DL) backend used. These tools support various reinforcement learning paradigms including Deep Reinforcement Learning (DRL), Model-Based Reinforcement Learning (MBRL), Multi-Agent Reinforcement Learning (MARL), and Contrastive Reinforcement Learning (CRL).

    Key functional areas within the common module include:

    • Tuning Tools: Utilities for hyperparameter tuning and optimization.
    • Callbacks: Mechanisms to hook into the training lifecycle.
    • Memory Tools: Specialized memory implementations for both single-agent (DRL) and multi-agent (MARL) reinforcement learning, including offline memory utilities.
    • Statistic Tools: Tools for collecting and analyzing training statistics.
    • General Utilities: A suite of common_tools, offline_util, and segtree_tool for general reinforcement learning tasks.

  7. Overview of XuanCe Reinforcement Learning Library

    master

    XuanCe is a comprehensive and unified Deep Reinforcement Learning (DRL) library. It provides a structured framework for various reinforcement learning paradigms, including:

    • DRL (Deep Reinforcement Learning): Single-agent reinforcement learning algorithms.
    • MBRL (Model-Based Reinforcement Learning): Algorithms that learn a model of the environment.
    • MARL (Multi-Agent Reinforcement Learning): Algorithms designed for multiple interacting agents.
    • CRL (Contrastive Reinforcement Learning): Constructive reinforcement learning approaches.
    • Offline RL: Learning from fixed datasets without environment interaction.

    The library supports multiple backends including PyTorch, TensorFlow, and MindSpore, and provides a wide range of environments such as Classic Control, Box2D, and MuJoCo.

  8. Overview of XuanCe Library

    master

    XuanCe (玄策) is an open-source ensemble of Deep Reinforcement Learning (DRL) algorithm implementations. It is designed to be a high-quality, modular, and easy-to-understand library that supports multiple deep learning backends including PyTorch, TensorFlow2, and MindSpore.

    Key features include:

    • Multi-Backend Support: Compatible with PyTorch, TensorFlow, and MindSpore across CPU, GPU, Linux, Windows, and MacOS.
    • Task Versatility: Supports DRL, Model-Based RL (MBRL), Multi-Agent RL (MARL), and Contrastive RL (CRL).
    • Scalability: Supports distributed training with multi-GPUs and fast execution via parallel environments.
    • Advanced Tooling: Built-in support for automatic hyperparameter tuning and visualization via Tensorboard or WandB.

  9. Explore Multi-Agent Reinforcement Learning (MARL) algorithms

    master

    XuanCe provides a wide range of Multi-Agent Reinforcement Learning (MARL) algorithms for solving multi-agent tasks. These algorithms are categorized into several families, including value decomposition, actor-critic, and communication-based methods.

    Available MARL algorithms include:

    Value Decomposition & Coordination:

    • IQL: Independent Q-Learning
    • VDN: Value Decomposition Networks
    • QMIX: Q-Mixing Networks
    • WQMIX: Weighted Q-Mixing Networks
    • QTRAN: Q-Transformation
    • DCG: Deep Coordination Graphs

    Actor-Critic & Policy Gradient:

    • IDDPG: Independent Deep Deterministic Policy Gradient
    • MADDPG: Multi-agent Deep Deterministic Policy Gradient
    • IAC: Independent Actor-Critic
    • COMA: Counterfactual Multi-agent Policy Gradient
    • VDAC: Value-Decomposition Actor-Critic
    • IPPO: Independent Proximal Policy Optimization
    • MAPPO: Multi-agent Proximal Policy Optimization
    • MFQ: Mean-Field Q-Learning
    • MFAC: Mean-Field Actor-Critic
    • ISAC: Independent Soft Actor-Critic
    • MASAC: Multi-agent Soft Actor-Critic
    • MATD3: Multi-agent Twin Delayed Deep Deterministic Policy Gradient

    Communication-based:

    • IC3Net: Individual Controlled Continuous Communication Model

  10. Create custom environments in XuanCe

    master

    XuanCe supports the integration of custom simulation or application scenarios through two primary environment types. This allows you to run XuanCe's built-in algorithms (such as DQN, PPO, IPPO, etc.) on your own tasks like robotics, trading, or custom multi-agent interactions.

    Depending on your task requirements, you can implement:

    1. Single-Agent Environment: Based on a standard Markov Decision Process (MDP).
    2. Multi-Agent Environment: Based on a Partial Observable Markov Decision Process (POMDP) structure.

    Once implemented, these environments can be used directly with XuanCe's reinforcement learning algorithms.

  11. Use Policy Gradient learners in XuanCe

    master

    XuanCe provides a variety of Policy Gradient-based learners within the xuance.torch.learners.policy_gradient module. These learners are designed for different reinforcement learning paradigms, including Actor-Critic, Off-Policy, and Maximum Entropy methods.

    Available learner modules include:

    • a2c_learner: Advantage Actor-Critic
    • ddpg_learner: Deep Deterministic Policy Gradient
    • mpdqn_learner: Multi-Priority Deep Q-Network (Policy Gradient variant)
    • npg_learner: Natural Policy Gradient
    • pdqn_learner: Policy-DQN
    • pg_learner: Basic Policy Gradient
    • ppg_learner: Proximal Policy Gradient
    • ppo_learner: Proximal Policy Optimization
    • ppokl_learner: PPO with KL divergence constraints
    • sac_learner: Soft Actor-Critic
    • sacdis_learner: Soft Actor-Critic for Discrete action spaces
    • spdqn_learner: Soft Policy Deep Q-Network
    • td3_learner: Twin Delayed Deep Deterministic Policy Gradient

  12. Available TensorFlow communication modules in XuanCe

    master

    XuanCe provides several specialized communication modules for Multi-Agent Reinforcement Learning (MARL) implemented using TensorFlow. These modules allow agents to exchange information, which is critical for coordination in multi-agent environments.

    Available modules include:

    • attention_comm: Communication based on attention mechanisms.
    • base_comm: Base communication implementations.
    • comm_net: Communication networks.
    • emergent_comm: Modules designed for emergent communication research.
    • gnn_comm: Graph Neural Network (GNN) based communication.
    • ic3net_comm: Implementation of IC3Net (Interacting Communicated Agents) communication protocols.