PokerRL Documentation

PokerRL provides four primary metrics for evaluating the performance of poker algorithms:

1. Best Response (BR): Computes the exact exploitability (intended for small games).
2. Local Best Response (LBR): Approximates a lower bound of BR; optimized for distributed computing in large games.
3. RL Best Response (RL-BR): Approximates BR by training a DDQN against the AI.
4. Head-To-Head (H2H): Measures performance by letting two modes of an agent play against each other.

For small games, unoptimized implementations of vanilla CFR, CFR+, and Linear CFR are also available and can be run like Deep RL agents, with results plotted to TensorBoard.

PokerRL uses ray to support scaling from local execution to distributed clusters. You can switch between these modes by changing a boolean flag in your TrainingProfile.

• Distributed Mode: Runs many worker processes on a single machine with many cores.
• Cluster Mode: Runs many workers across many different machines.

To use Cluster mode, you must provide a Ray cluster specification file (.yaml) and have your AWS account configured.

PokerRL is a framework for Multi-Agent Deep Reinforcement Learning in poker games. The architecture is built around several key abstractions:

• Workers: Agents that interact with each other to perform training.
• TrainingProfile: An instance (found in .../rl/base_cls/TrainingProfileBase) used to pass arguments for a training run.
• EvalAgent: A wrapper for a trained agent (found in .../rl/base_cls/EvalAgentBase).
• AgentTournament: A component used to have multiple EvalAgent instances battle each other.
• InteractiveGame: A component used to play against humans (found in .../game/InteractiveGame).

While the game engine and agent modules support $N > 1$ players, some specific parts of the framework are optimized for 2-player games. Local workers can be wrapped with approximately 4 lines of code to function as independent distributed workers.

You can verify your installation by running the built-in unit tests or by running the provided examples.

Run Unit Tests:

python -m unittest discover PokerRL/test

Run Interactive/Training Examples:

• Play poker against yourself: examples/interactive_user_v_user.py
• Train a CFR+ agent in a small game: examples/run_cfrp_example.py

To install PokerRL on your local machine, ensure you have Anaconda/Miniconda and Docker installed.

Note: For distributed runs, you must use Linux and install the distributed extra: pip install PokerRL[distributed]. This is not required for local-only usage.

Follow these steps to set up a Python 3.6 environment with PyTorch:

# Create and activate environment
conda create -n CHOOSE_A_NAME python=3.6 -y
source activate THE_NAME_YOU_CHOSE

# Install dependencies
pip install requests
conda install pytorch=0.4.1 -c pytorch
pip install PokerRL

PokerRL uses PyCrayon (a wrapper around Tensorboard) for logging. To set up the log server, use Docker to run the alband/crayon image:

```bash
docker run -d -p 8888:8888 -p 8889:8889 --name crayon alband/crayon
docker start crayon

After running these commands, you can access Tensorboard in your browser at localhost:8888.