PettingZoo Documentation

The SISL environments are cooperative multi-agent benchmarks originally released as part of "Cooperative multi-agent control using deep reinforcement learning."

Note: PettingZoo includes major bug fixes for these environments. Users are discouraged from directly comparing results obtained with these versions to the results reported in the original paper.

PettingZoo provides several families of environments:

• Atari: Multi-player Atari 2600 games (cooperative, competitive, and mixed sum).
• Butterfly: Cooperative graphical games requiring high coordination.
• Classic: Classical games including card games and board games.
• SISL: Two cooperative environments originally from the SISL MADRL repository.

The following training patterns are available when using AgileRL with PettingZoo environments:

• DQN: Training a DQN agent using curriculum learning and self-play (e.g., Connect Four).
• MADDPG: Training a Multi-Agent Deep Deterministic Policy Gradient agent (e.g., Multi-agent Atari games or co-operative simple speaker listener environments).
• MATD3: Training a Multi-Agent Twin Delayed Deep Deterministic Policy Gradient agent (e.g., multi-particle-environment games).

LangChain is a framework for developing applications powered by language models through composability. It focuses on six main functional areas:

• LLMs and Prompts: Prompt management, optimization, and a generic interface for all LLMs.
• Chains: Sequences of calls (to LLMs or other utilities) using a standard interface.
• Data Augmented Generation: Chains that interact with external data sources to fetch context before generation (e.g., question-answering over specific data).
• Agents: LLMs that make decisions about which Actions to take, observe the resulting Observation, and repeat the process.
• Memory: Persisting state between calls of a chain or agent.
• Evaluation: Using language models to evaluate the outputs of other generative models.

PettingZoo provides two primary API standards depending on the nature of your multi-agent reinforcement learning (MARL) problem:

1. AEC API: Supports sequential, turn-based environments where agents act one after another.
2. Parallel API: Supports environments where agents perform actions simultaneously.

This tutorial demonstrates how to use the Tianshou library to train a Deep Q-Network (DQN) agent to compete against a random policy agent within the PettingZoo tictactoe environment.

To implement this, you typically need to:

1. Set up the PettingZoo environment.
2. Define the Tianshou policy (e.g., DQN).
3. Configure the training loop using Tianshou's buffers and learners.

Use api_test to ensure an AEC (Agent Environment Cycle) environment is consistent with the PettingZoo API. The test will raise an error if API requirements are not met or return normally if successful.

Arguments:

• num_cycles (int): Number of cycles to run to check API consistency.
• verbose_progress (bool): If True, prints progress messages, which is useful for debugging.

from pettingzoo import make
from pettingzoo.test import api_test

env = make("aec", "butterfly/pistonball-v6")
api_test(env, num_cycles=1000, verbose_progress=False)

Use seed tests to ensure that calling seed() makes the environment deterministic. This is critical for reproducible evaluations.

AEC Environments: Use seed_test(env_factory_function). Parallel Environments: Use parallel_seed_test(parallel_env_factory_function).

Arguments:

• num_cycles (int): How long to run the environment to check for determinism.
• test_kept_state (bool): If False, disables the second test (which checks if a single environment remains deterministic after seed() then reset()). This is useful for physics-based environments that may have minor non-deterministic artifacts due to caches.

from pettingzoo.butterfly.pistonball.pistonball import env, parallel_env
from pettingzoo.test import seed_test, parallel_seed_test

# For AEC
seed_test(env, num_cycles=10)

# For Parallel
parallel_seed_test(parallel_env)