OpenAI Baselines

When evaluating the performance of GAIL or BC imitators on Mujoco environments, results are categorized by policy type:

• Deterministic Policy: Achieved by setting the standard deviation (std) to 0.
• Stochastic Policy: Uses the default stochastic behavior of the policy.

Performance is reported in both Un-normalized and Normalized score formats across environments like Hopper-v1, HalfCheetah-v1, Walker2d-v1, Humanoid-v1, and HumanoidStandup-v1.

The current implementation of ACKTR has been refactored to handle both discrete and continuous action spaces uniformly within the same codebase.

Note on Performance: If you require the original implementation for continuous action spaces (which may perform better on MuJoCo tasks), you must use the old_acktr_cont branch. The refactored version is the default in the main repository.

You can quickly train a Deep Q-Network (DQN) agent on the CartPole-v0 environment and then visualize its performance using the baselines.run module.

1. Train the model: Run the training command specifying the algorithm (deepq), environment (CartPole-v0), a save path, and the number of timesteps.
2. Visualize the policy: Load the saved .pkl file using the --load_path flag and use the --play flag to watch the agent interact with the environment.

# Train model and save the results to cartpole_model.pkl
python -m baselines.run --alg=deepq --env=CartPole-v0 --save_path=./cartpole_model.pkl --num_timesteps=1e5

# Load the model saved in cartpole_model.pkl and visualize the learned policy
python -m baselines.run --alg=deepq --env=CartPole-v0 --load_path=./cartpole_model.pkl --num_timesteps=0 --play

A script is provided to generate the demonstration dataset specifically for the FetchPickAndPlace task. This outputs a .npz file used for training.

python experiment/data_generation/fetch_data_generation.py

To train an agent using Generative Adversarial Imitation Learning (GAIL), you must first provide expert demonstrations.

1. Download expert data: Download the required expert data and place it in the ./data directory of your project.
2. Execute GAIL: Run the training script using the baselines.gail.run_mujoco module.

You can run the training on a single rank or distribute it across multiple ranks using MPI.

# Run with single rank
python -m baselines.gail.run_mujoco

# Run with multiple ranks (e.g., 16 ranks)
mpirun -np 16 python -m baselines.gail.run_mujoco

To save the trained policy, use the --save_path flag. To inspect the learned behavior of an agent, use the --play flag. The --play flag can be combined with --load_path to load a specific trained policy for evaluation.

# Save a policy
python -m baselines.run --alg=her --env=FetchReach-v1 --num_timesteps=5000 --save_path=~/policies/her/fetchreach5k

# Play/Inspect a policy
python -m baselines.run --alg=her --env=FetchReach-v1 --num_timesteps=5000 --play

To verify your installation, install pytest and run the test suite:

pip install pytest
pytest

To install OpenAI Baselines, follow these steps:

1. Install System Prerequisites:

   • Ubuntu: sudo apt-get update && sudo apt-get install cmake libopenmpi-dev python3-dev zlib1g-dev
   • Mac OS X: Use Homebrew: brew install cmake openmpi

2. Set up a Virtual Environment (Recommended):

   pip install virtualenv
   virtualenv /path/to/venv --python=python3
   . /path/to/venv/bin/activate

3. Install TensorFlow: The master branch supports TensorFlow 1.4 to 1.14. For TensorFlow 2.0 support, use the tf2 branch.

   • For GPU: pip install tensorflow-gpu==1.14
   • For CPU: pip install tensorflow==1.14

4. Install Baselines:

   git clone https://github.com/openai/baselines.git
   cd baselines
   pip install -e .

For advanced users looking to customize the DQN algorithm or train on Atari environments:

• Fine-grained control: Refer to baselines/deepq/experiments/custom_cartpole.py to see how to control the internals of the DQN algorithm beyond the standard deepq.learn interface.
• Atari training: Use baselines/deepq/defaults.py for settings optimized for Atari. To train on Atari Pong, use the following command:

python -m baselines.run --alg=deepq --env=PongNoFrameskip-v4

To reproduce parallelized training results (e.g., from Plappert et al. 2018), use mpirun to launch multiple MPI processes. This requires a machine with sufficient physical CPU cores to handle the parallel trajectories.

mpirun -np 19 python -m baselines.run --num_env=2 --alg=her ... 

Use the --save_path and --load_path options to manage TensorFlow state.

1. Save a model during training:

   python -m baselines.run --alg=ppo2 --env=PongNoFrameskip-v4 --num_timesteps=2e7 --save_path=~/models/pong_20M_ppo2

2. Load and visualize a model: To visualize what a model has learned, load the path, set timesteps to 0, and use the --play flag:

   python -m baselines.run --alg=ppo2 --env=PongNoFrameskip-v4 --num_timesteps=0 --load_path=~/models/pong_20M_ppo2 --play

Note on MuJoCo: MuJoCo environments require normalization. To ensure models are portable, normalization coefficients are saved as TensorFlow variables. If you require high-throughput steps and do not need to save/restore, you can set use_tf=False in baselines/run.py to use numpy normalization.

HER can be trained using pre-recorded demonstrations by providing a demonstration file via the --demo_file flag. This incorporates behavior cloning loss as an auxiliary loss to help overcome exploration problems.

python -m baselines.run --alg=her --env=FetchPickAndPlace-v1 --num_timesteps=2.5e6 --demo_file=/Path/to/demo_file.npz