Stable Baselines

DQN implements the Deep Q-Network algorithm and includes extensions like Double-DQN, Dueling-DQN, and Prioritized Experience Replay.

Important Constraints:

• Policy Models: DQN does not support stable_baselines.common.policies. You must use its specific policy models (e.g., MlpPolicy, CnnPolicy).
• Recurrent Policies: Not supported.
• Multi-processing: Not supported.
• Gym Spaces:
  • Action Space: Must be Discrete.
  • Observation Space: Supports Discrete and Box.

Default Extensions:

• Double-Q learning and Dueling DQN are enabled by default.
• To disable Dueling DQN, refer to GitHub Issue #406.
• To disable Double-Q learning, pass the appropriate value to the constructor.

from stable_baselines import DQN
from stable_baselines.deepq.policies import MlpPolicy
import gym

env = gym.make('CartPole-v1')
model = DQN(MlpPolicy, env, verbose=1)
model.learn(total_timesteps=25000)
model.save("deepq_cartpole")

TD3 is an actor-critic algorithm that improves upon DDPG using clipped double Q-learning, delayed policy updates, and target policy smoothing.

Important Constraints:

• Policies: TD3 does not support stable_baselines.common.policies. It must use its own specific policy models (e.g., MlpPolicy, CnnPolicy) because it requires double Q-value estimation.
• Action Space: Supports Box (Continuous) and Discrete (though typically used for continuous control). It does not support MultiDiscrete or MultiBinary action spaces.
• Observation Space: Supports Box, Discrete, MultiDiscrete, and MultiBinary observation spaces.
• Limitations: Does not support recurrent policies or multi-processing.

import gym
from stable_baselines import TD3
from stable_baselines.td3.policies import MlpPolicy
from stable_baselines.ddpg.noise import NormalActionNoise
import numpy as np

env = gym.make('Pendulum-v0')
n_actions = env.action_space.shape[-1]
action_noise = NormalActionNoise(mean=np.zeros(n_actions), sigma=0.1 * np.ones(n_actions))

model = TD3(MlpPolicy, env, action_noise=action_noise, verbose=1)
model.learn(total_timesteps=50000)
model.save("td3_pendulum")

Algorithm selection depends primarily on your Action Space and whether you need Multiprocessing.

Discrete Actions

(Includes Discrete, MultiDiscrete, Binary, and MultiBinary)

• Single Process: Use DQN (and its extensions like double DQN or prioritized replay) or ACER. DQN is generally more sample efficient due to its replay buffer.
• Multiprocessed: Use PPO2, A2C, or their successors (ACKTR, ACER).
• MPI Support: If you can use MPI for multiprocessing, consider PPO1 or TRPO.

Continuous Actions

• Single Process: SAC and TD3 are the current State Of The Art (SOTA). Note: Normalization is critical for these algorithms.
• Multiprocessed: Use PPO2, TRPO, or A2C. If using MPI, you can also choose PPO1 or DDPG.

Goal Environments

• If your environment implements the GoalEnv interface, use HER (Hindsight Experience Replay) combined with SAC, TD3, DDPG, or DQN depending on the action space.

PPO1 (Proximal Policy Optimization) is an implementation that combines ideas from A2C and TRPO. It uses clipping to ensure the new policy does not deviate too far from the old policy during updates.

Key Requirements & Differences:

• MPI Requirement: PPO1 requires OpenMPI. If OpenMPI is not enabled, PPO1 will not be imported into the stable_baselines module.
• Multiprocessing: Unlike PPO2 (which uses vectorized environments), PPO1 uses MPI for multiprocessing.
• Recurrent Policies: Not supported (❌).

Supported Gym Spaces:

import gym
from stable_baselines.common.policies import MlpPolicy
from stable_baselines import PPO1

env = gym.make('CartPole-v1')

model = PPO1(MlpPolicy, env, verbose=1)
model.learn(total_timesteps=25000)
model.save("ppo1_cartpole")

# Loading and using the model
model = PPO1.load("ppo1_cartpole")
obs = env.reset()
while True:
    action, _states = model.predict(obs)
    obs, rewards, dones, info = env.step(action)
    env.render()