ElegantRL Documentation

QMIX is a value-based method for Deep Multi-Agent Reinforcement Learning that enables training decentralized policies in a centralized end-to-end fashion. It uses a network to estimate joint action-values as a non-linear combination of per-agent values based on local observations.

Supported Features:

• Experience replay: Yes
• Target network: Yes
• Prioritized Experience Replay (PER): Yes
• Gradient clipping: No
• Reward clipping: No
• Ornstein–Uhlenbeck noise: No

RLSolver is a high-performance Reinforcement Learning (RL) solver designed to find high-quality or nearly global optima for both nonconvex/nonlinear continuous optimization problems and combinatorial discrete optimization problems. It provides pretrained neural networks for real-time inference in these domains.

Key technical features include:

• Massively parallel simulations: Uses GPU (CUDA and Tensor cores) to simulate gym-environments.
• Stability via Hamiltonian-term: Implements the Hamiltonian-term as an add-on regularization to Deep Reinforcement Learning (DRL) algorithms, utilizing GPU parallel computing to search for 'minimum-energy states'.
• Problem Domains: Supports MCMC-related tasks such as Maxcut, MIMO Beamforming (5G/6G), Classical NP-Hard problems, Quantum Circuit simulation, Compressive Sensing, Portfolio Management, and OR-Gym.

ElegantRL implements several model-free Deep Reinforcement Learning (DRL) algorithms:

• Continuous actions (single-agent): DDPG, TD3, SAC, PPO, REDQ
• Discrete actions (single-agent): DQN, Double DQN, D3QN
• Multi-agent environment: QMIX, VDN, MADDPG, MAPPO, MATD3

Supported simulators:

• Isaac Gym: For massively parallel simulations.
• Benchmarking: OpenAI Gym, MuJoCo, PyBullet, FinRL.

ElegantRL provides implementations for several Deep Reinforcement Learning algorithms categorized by action space and agent type:

Continuous Actions

• DDPG, TD3, SAC, A2C, PPO, REDQ

Discrete Actions

• DQN, DoubleDQN, D3QN, PPO-Discrete

Multi-Agent RL

• QMIX, VDN
• MADDPG, MAPPO, MATD3

ElegantRL supports multiple levels of parallelism:

• Agent Parallelism: Training hundreds of agents in parallel via Population-Based Training (PBT), useful for ensemble methods.
• Worker Parallelism: Accelerates data collection by generating transitions in parallel. Supported methods include:
  • Using a VecEnv to generate transitions in batches.
  • Using multiple workers for environments that are not VecEnv.
• Learner Parallelism: Training multiple critics and multiple actors in parallel for ensemble DRL methods to improve stability and reduce overestimation bias.

ElegantRL is designed as a cloud-native framework for massively parallel Deep Reinforcement Learning (DRL) training. It utilizes a microservice-based architecture to handle the full DRL lifecycle (simulate-learn-deploy) on cloud computing power.

Key architectural components include:

• Microservices: DRL agents are organized into specialized microservices such as orchestrator, worker, learner, and evaluator, which communicate via APIs.
• Containerization: Each microservice is mapped to a container, and the entire agent is encapsulated into a Kubernetes (K8s) Pod.
• Orchestration: The framework uses Kubernetes to automate the deployment and management of DRL applications, providing benefits like asynchronous parallelism, fault isolation, and security.

TD3 is an improvement over DDPG that uses Clipped Double-Q Learning, Delayed Policy Updates, and Target Policy Smoothing to prevent Q-value overestimation.

Supported features in this implementation:

• Experience replay
• Target network
• Gradient clipping
• Prioritized Experience Replay (PER)

Note: The implementation uses a CriticTwin class which shares parameters between two Q-networks to reduce computational and training time costs.

Warning: TD3 contains highly sensitive hyperparameters that require careful tuning for satisfactory results.

The ElegantRL HelloWorld project follows a 'Net-Agent-Env-Run' architecture to implement Deep Reinforcement Learning (DRL). The core components are:

• net.py: Defines the neural network architectures, including Q-Net, Actor Network, and Critic Network.
• agent.py: Contains DRL algorithm implementations (e.g., DQN, SAC, PPO) and the ReplayBuffer class for storing transitions.
• erl_env.py: Provides a wrapper class to preprocess Gym-styled environments.
• run.py: Orchestrates the training and evaluation loop via the train_and_evaluate process.
• demo.py: Provides entry points for running specific algorithm/action-space combinations.

ElegantRL supports three ensemble methods:

1. Weighted average
2. Model fusion
3. Tournament-based ensemble training scheme

Tournament-based ensemble training is a cloud orchestration mechanism that schedules interactions between a leaderboard and a training pool containing hundreds of agents (pods). This is detailed in the ElegantRL-Podracer documentation.