pymarl3

The Hyper Policy Network (HPN) is a general-purpose architecture designed to equip Multi-Agent Reinforcement Learning (MARL) algorithms with permutation invariance (PI) and permutation equivariance (PE) properties. It uses hypernetworks to generate customized weights for different input components, ensuring that the same input always receives the same weight, regardless of order.

Key components include:

• Permutation Invariant Input Layer: A shared hypernetwork takes an input set $X_j$ (a batch of $m$ inputs of dimension $k$) and generates unique weights $W_i$ for each input $x_i$. The outputs are merged using a permutation-invariant 'sum' function.
• Permutation Equivariance Output Layer: To maintain equivariance, the weights and biases for agent-related actions (e.g., $\mathcal{A}_i^{\text{attack}}$) are also generated via a hypernetwork. Because the hypernetwork maintains a one-to-one mapping between inputs and weights, changing the input order results in the same change in output order.

HPN can be integrated into existing algorithms like VDN, QMIX, MADDPG, and MAPPO to improve learning speed and performance.

API-HPN (Agent Permutation Invariant Hypernetwork) is a neural architecture designed to boost Multi-Agent Reinforcement Learning (MARL) by exploiting permutation invariance and equivariance.

In MARL environments with homogeneous components (like agents or NPCs), the order of inputs shouldn't change the underlying information. API-HPN addresses this using two main modules:

1. Agent Permutation Invariant Input Layer: Uses a shared hypernetwork to generate unique weights $W_i$ for each input component $x_i$. Each component is processed by its specific weight, and the results are merged using a permutation-invariant 'sum' function. This ensures that shuffling the input components does not change the output.
2. Agent Permutation Equivariance Output Layer: For outputs that have a one-to-one correspondence with input components (e.g., attack actions targeting specific enemies), a hypernetwork generates weights such that permuting the inputs results in the same permutation of the outputs.

API-HPN is designed to be a general-purpose component that can be integrated into existing MARL algorithms like VDN, QMIX, MADDPG, and MAPPO to reduce sample complexity and improve convergence.

To use API-HPN effectively, it is important to distinguish between the two types of permutation properties used in the architecture:

• Permutation Invariant Function: A function where permuting the input components does not change the output. Mathematically: $f([x_1, x_2, …, x_m]^T) = f(M[x_1, x_2, …, x_m]^T)$, where $M$ is a permutation matrix.
• Permutation Equivariant Function: A function where permuting the input components results in the same permutation of the output components. Mathematically: $f(M[x_1, x_2, …, x_m]^T) = M[y_1, y_2, …, y_m]^T$.

Application Example (SMAC): In the StarCraft II micromanagement benchmark (SMAC), an agent's observation can be split into an ally group and an enemy group.

• Equivariance: Q-values for attack actions (which correspond to specific enemies) should be equivariant to permutations of the enemy group.
• Invariance: Q-values for move actions (which are general to the agent) should be invariant to the permutations of the entire set.

To reproduce experimental results, use the src/main.py script. You must specify an algorithm configuration using --config and an environment configuration using --env-config.

Configuration File Locations:

• --config targets files located in src/config/algs.
• --env-config targets files located in src/config/envs.

Example: Running VDN on SMAC (5m_vs_6m scenario)

CUDA_VISIBLE_DEVICES="0" python src/main.py --config=api_vdn --env-config=sc2 with env_args.map_name=5m_vs_6m obs_agent_id=True obs_last_action=False runner=parallel batch_size_run=8 buffer_size=5000 t_max=10050000 epsilon_anneal_time=100000 batch_size=128 td_lambda=0.6

Example: Running QMIX on SMAC (5m_vs_6m scenario)

CUDA_VISIBLE_DEVICES="1" python src/main.py --config=api_qmix --env-config=sc2 with env_args.map_name=5m_vs_6m obs_agent_id=True obs_last_action=False runner=parallel batch_size_run=8 buffer_size=5000 t_max=10050000 epsilon_anneal_time=100000 batch_size=128 td_lambda=0.6

# For SMAC, take the 5m_vs_6m scenario for example.
CUDA_VISIBLE_DEVICES="0" python src/main.py --config=api_vdn --env-config=sc2 with env_args.map_name=5m_vs_6m obs_agent_id=True obs_last_action=False runner=parallel batch_size_run=8 buffer_size=5000 t_max=10050000 epsilon_anneal_time=100000 batch_size=128 td_lambda=0.6

CUDA_VISIBLE_DEVICES="1" python src/main.py --config=api_qmix --env-config=sc2 with env_args.map_name=5m_vs_6m obs_agent_id=True obs_last_action=False runner=parallel batch_size_run=8 buffer_size=5000 t_max=10050000 epsilon_anneal_time=100000 batch_size=128 td_lambda=0.6

You can reproduce results for various scenarios (e.g., 5m_vs_6m, 3s5z_vs_3s6z, 6h_vs_8z, etc.) using src/main.py.

Common arguments used in these commands include:

• --config: Specifies the model configuration (e.g., hpn_qmix, qmix, hpn_qplex, qplex).
• --env-config: Specifies the environment configuration (e.g., sc2).
• with: Used to override specific hyperparameter values in the config.
• env_args.map_name: Sets the specific StarCraft II map.
• obs_agent_id: Boolean to include agent ID in observations.
• obs_last_action: Boolean to include last action in observations.
• runner: Specifies the runner type (e.g., parallel).
• batch_size_run: Number of environments to run in parallel.
• buffer_size: Size of the replay buffer.
• t_max: Total training steps.
• epsilon_anneal_time: Time steps for epsilon annealing.
• batch_size: Training batch size.
• td_lambda: TD lambda value.

Note: For some maps like 6h_vs_8z or 3s_vs_5z, additional parameters like hpn_head_num may be required.

# Example for 5m_vs_6m using hpn_qmix
CUDA_VISIBLE_DEVICES="0" python src/main.py --config=hpn_qmix --env-config=sc2 with env_args.map_name=5m_vs_6m obs_agent_id=True obs_last_action=False runner=parallel batch_size_run=8 buffer_size=5000 t_max=10050000 epsilon_anneal_time=100000 batch_size=128 td_lambda=0.6