DOCSEARCHbyAlgolia
Submit Docs

OpenSpiel: A Framework for Reinforcement Learning in Games

repository·master·Indexed 26 days ago

https://github.com/google-deepmind/open_spiel

OpenSpiel is a comprehensive framework for reinforcement learning and planning in games, providing a collection of environments and algorithms for research. It includes support for various game types, including extensive-form games in Gambit format, mean field games, and specific implementations like Hive, Slovenian Tarok, and Hanabi. The framework also provides a C++ LibTorch-based AlphaZero implementation and interfaces for GAMUT and Universal Poker.

Tokens
45.8K
Snippets
134
Records
214
Agent score
90%

What's inside OpenSpiel

  1. Overview of available algorithms in OpenSpiel

    master

    OpenSpiel provides a wide range of algorithms categorized by their application in game theory and reinforcement learning. Algorithms are marked with the following status indicators:

    • ✅ Thoroughly-tested: Verified against known values or reproduced from papers.
    • 🔶 Implemented but lightly tested: Functional but may lack extensive verification.
    • ❌ Known problems: Use with caution; check GitHub issues for known bugs.

    Algorithm Categories

    Search

    Used for finding optimal moves in games (e.g., Minimax, MCTS, IS-MCTS).

    Optimization (Opt.)

    Used for finding equilibria (e.g., Lemke-Howson, Sequence-form LP, Shapley Values).

    Tabular

    Algorithms that operate on discrete state/action spaces (e.g., CFR, Q-learning, SARSA, Regret Matching).

    Reinforcement Learning (RL)

    Deep RL algorithms (e.g., DQN, PPO, A2C).

    Multi-Agent Reinforcement Learning (MARL)

    Algorithms designed for multiple interacting agents (e.g., AlphaZero, NFSP, PSRO, LOLA).

    Evaluation and Visualization (Eval. / Viz.)

    Tools for analyzing game dynamics and equilibrium quality (e.g., $\alpha$-Rank, Nash Averaging, Replicator Dynamics).

  2. Overview of Mean Field Games (MFG) Python API

    master

    The Mean Field Games (MFG) Python API in OpenSpiel is designed for modeling interactions in very large populations where the number of agents $N$ approaches infinity ($N \to \infty$). Instead of tracking individual agents, the API tracks the probability distribution (the 'population state').

    Note: This API is currently experimental. Use it only if you are part of the project. For finite-agent interactions, use the main OpenSpiel API.

  3. Overview of OpenSpiel

    master

    OpenSpiel is a framework for research in general reinforcement learning and search/planning in games. It supports a wide variety of game types, including:

    • n-player (single- and multi-agent) zero-sum, cooperative, and general-sum games.
    • One-shot and sequential games.
    • Strictly turn-taking and simultaneous-move games.
    • Perfect and imperfect information games.
    • Traditional multiagent environments like grid worlds (partially and fully observable) and social dilemmas.

    Games are represented as procedural extensive-form games. The core API and games are implemented in C++ and exposed to Python, while algorithms and tools are available in both C++ and Python.

  4. Overview of Slovenian Tarok implementation

    master

    Slovenian Tarok is a three- or four-player, trick-taking, competitive game of skill with bidding. The OpenSpiel implementation is a full game but currently excludes announcements (which are considered an optional addition).

    Key Technical Detail: The environment is implemented in an implicitly stochastic manner. Instead of a sequence of chance nodes for dealing cards, the chance node returns a single dummy action that utilizes an internal RNG to deal all cards at once. This design is intended to simplify implementation and accommodate the large game tree size.

  5. Overview of OpenSpiel capabilities

    master

    OpenSpiel is a framework for reinforcement learning and search/planning research in games. It represents games as procedural extensive-form games and supports a wide variety of game types, including:

    • Player counts: Single and multi-player games.
    • Information: Fully observable (via observations) and imperfect information (via information states and observations).
    • Stochasticity: Explicit chance nodes (and partial implicit stochasticity).
    • Game Forms: n-player normal-form 'one-shot' games and (2-player) matrix games.
    • Move Types: Sequential and simultaneous move games.
    • Payoff Structures: Zero-sum, general-sum, and cooperative (identical payoff) games.

  6. Overview of Voting as Evaluation (VasE) implementation

    master
    This module provides a general implementation of voting rules from computational social choice. Specifically, it implements the voting rules used in 'Voting as Evaluation (VasE)' as described by Lanctot et al. in Evaluating Agents using Social Choice Theory. The implementation includes examples of running VasE on Atari datasets referenced in the paper.

  7. Understand the Hive game implementation and UHP compatibility

    master

    The Hive implementation includes the base game and three expansion pieces: Mosquito, Ladybug, and Pillbug.

    This implementation follows the Universal Hive Protocol (UHP). This allows game states to be serialized to and deserialized from valid UHP game strings. Consequently, this implementation can be used as a UHP-compliant Hive Engine for interacting with other engines.

  11. Browse available games in OpenSpiel

    master

    OpenSpiel includes a wide variety of games categorized by their characteristics. The documentation uses icons to indicate game status:

    • 🟢 Green: Fully supported/stable.
    • 🔶 Orange: Partially supported or specific implementation details apply.
    • ❌ Red: Currently unavailable or broken (check associated GitHub issues).

    Common game types available include:

    Imperfect Information & Hidden Information

    • Phantom Go: Go variant where opponent's stones are hidden.
    • Phantom Tic-Tac-Toe: Tic-tac-toe with hidden opponent tokens.
    • Reconnaissance Blind Chess: Chess with unknown opponent pieces and sensing moves.
    • Social Deduction: Asymmetric information with discrete communication.

    Classic Board & Card Games

    • Tic-Tac-Toe: Standard 2-player pattern game.
    • Nim: 2-player object-taking game.
    • Shogi: Chess-like game with piece dropping mechanics.
    • Oware: Token redistribution game.
    • Pentago: Token placement with board rotation.
    • Poker (Hold 'em): Implemented via ACPC.

    Multi-Agent & Economic Games

    • Negotiation: Agents negotiating resource allocation.
    • Prisoner's Dilemma: Classic game theory payoff scenario.
    • Mean Field Game (routing): Large-scale routing with congestion functions.
    • Sheriff: Bargaining game useful for studying correlated equilibria.

    Single Player & Pathfinding

    • Morpion Solitaire (4D): Maximizing lines on a grid.
    • Pathfinding: Agents moving to destinations.

  12. Choose between Flax Linen and Flax NNX APIs

    master

    The AlphaZero framework supports two different neural network API paradigms via flax:

    1. flax.linen (found in model_linen.py): The currently stable API which follows a functional programming paradigm.
    2. flax.nnx (found in model_nnx.py): An experimental API that is closer to the Object-Oriented Programming (OOP) paradigm and provides lifted transforms.