InforMARL: Scalable Multi-Agent Reinforcement Learning Framework

Understanding the module layout for integration or customization:

• make_env.py: Provides code for importing a multiagent environment as an OpenAI Gym-like object.
• ./multiagent/environment.py: Core simulation logic, including interaction physics and the _step() function.
• ./multiagent/core.py: Defines base classes for Entities, Landmarks, Agents, etc.
• ./multiagent/rendering.py: Handles visual display of agent behaviors.
• ./multiagent/policy.py: Contains interactive policies driven by keyboard input.
• ./multiagent/scenario.py: The base scenario object used for all specific scenarios.
• ./multiagent/scenarios/: Directory containing individual scenario implementations.

The Surviving environment is a large-scale multi-agent environment designed to evaluate DGN.

Environment Specifications:

• Scale: 100 agents interacting in a grid-based world.
• Observations: Each agent has a limited local observation consisting of a 3x3 grid centered on the agent.
• Communication: Agents can communicate with neighbors within a 7x7 grid region.
• Actions: At each timestep, an agent can move to one of four neighboring grids or eat food at its current location.
• Mechanics:
  • Agents start with 10 health.
  • Agents lose 1 health per step.
  • Eating food increases health.
• Rewards:
  • If health reaches 0: -0.2 reward.
  • If agent survives: 0.4 reward.

To perform curriculum training, you must first specify the desired number of agents within the automate.py file. Once configured, execute the training using automate.py.

# Execute curriculum training
python automate.py --env-name simple_spread --entity-mp --save-dir 0

To switch from the GPG baseline to the VPG (VPG) baseline, modify main.py as follows:

1. Uncomment line 23.
2. Comment out line 22.

To observe agent formations using previously trained models, run parameter_reload.py. This script automatically loads model parameters from the ./logs directory.

python parameter_reload.py

To set up the environment, create a virtual environment using Python 3.5 and install the dependencies listed in requirements.txt using pip.

# Create a virtualenv with python 3.5 and install dependencies
pip install -r requirements.txt

To run the GPG (Graph Policy Gradient) baseline, you must install the following dependencies:

1. Deep Graph Library (DGL): Visit https://www.dgl.ai/ for installation instructions.
2. PyTorch: Ensure you have a version compatible with your installed DGL version.
3. gym_formation: You must install gym_formation following the specific instructions located within its own repository/folder before running GPG.

You can continue training from a previously saved model to transfer learned behaviors to a team with a different number of agents. Use the --continue-training flag along with --load-dir pointing to the saved model file.

# Example: Transferring a policy trained with 3 agents to a team of 5 agents in simple_spread
python main.py --env-name simple_spread --entity-mp --continue-training --load-dir models/ss/na3_uc.pt --num-agents 5

Gym Flock provides the FormationFlying-v3 environment, which is compatible with the standard OpenAI Gym API. You can interface with it using env.reset() and env.step().

Additionally, these implementations include an env.controller() method, which returns the best current set of actions suitable for imitation learning tasks.

import gym
import gym_flock

# Initialize the environment
env = gym.make("FormationFlying-v3")

# Standard Gym interface
obs = env.reset()
# ... in your loop ...
# action = agent.get_action(obs)
# next_obs, reward, done, info = env.step(action)

# For imitation learning, use the built-in controller
# best_actions = env.controller()

To train the InforMARL model, run the onpolicy/scripts/train_mpe.py script with the required arguments. Key arguments include --env_name (e.g., GraphMPE), --algorithm_name (e.g., rmappo), and --scenario_name (e.g., navigation_graph).

python -u onpolicy/scripts/train_mpe.py --use_valuenorm --use_popart \
--project_name "informarl" \
--env_name "GraphMPE" \
--algorithm_name "rmappo" \
--seed 0 \
--experiment_name "informarl" \
--scenario_name "navigation_graph" \
--num_agents 3 \
--collision_rew 5 \
--n_training_threads 1 --n_rollout_threads 128 \
--num_mini_batch 1 \
--episode_length 25 \
--num_env_steps 2000000 \
--ppo_epoch 10 --use_ReLU --gain 0.01 --lr 7e-4 --critic_lr 7e-4 \
--user_name "marl" \
--use_cent_obs "False" \
--graph_feat_type "relative" \
--auto_mini_batch_size --target_mini_batch_size 128