repository·master·Indexed 42 days ago
https://github.com/proroklab/popgymA collection of partially-observable procedural gym environments (version 1.0.7) designed to benchmark memory in deep reinforcement learning. It provides a suite of POMDP environments across game, navigation, diagnostic, and control categories, along with 13 memory model baselines implemented via RLlib, including LSTM, GRU, and Differentiable Neural Computers.
To create a custom memory model compatible with the POPGym/RLlib workflow, you must:
popgym.baselines.ray_models.base_model.initial_state function.memory_forward function.MODEL_CONFIG attribute.Once implemented, you can use your model in RLlib by passing it as the custom_model in your configuration dictionary.
You can install POPGym using pip. Depending on your needs, you can install the base package or include extra dependencies for navigation or baselines.
numpy and gymnasium.mazelib.rllib. Warning: Baselines rely on difficult-to-maintain, older dependencies and may require downgrading Python or other packages.# Install base environments
pip install popgym
# Include navigation environments (requires mazelib)
pip install "popgym[navigation]"
# Include baselines (requires rllib and older dependencies)
pip install "popgym[baselines]"To include the baselines (memory models) and RLlib support, install the [baselines] extra.
pip install popgym[baselines]To create a partially observable environment, instantiate an environment class (e.g., popgym.envs.position_only_cartpole.PositionOnlyCartPoleEasy) and apply wrappers to modify its observability or action space.
Common wrappers include:
PreviousAction: Conditions the observation on the previous action. Returns a zero action at the initial timestep by default.Antialias: Adds an indicator to the observation space to distinguish the initial timestep from subsequent timesteps when using PreviousAction.Flatten: Flattens nested observation and action spaces into a single space.DiscreteAction: Converts the action space into a single large Discrete space (useful for DQN-style algorithms).Markovian: Adds the hidden Markov state to the environment, which can be configured via the Observability enum.import popgym
from popgym.wrappers import PreviousAction, Antialias, Flatten, DiscreteAction, Markovian
from popgym.core.observability import Observability
# 1. Create base environment
env = popgym.envs.position_only_cartpole.PositionOnlyCartPoleEasy()
# 2. Add wrappers for POMDP conditioning and compatibility
wrapped_env = PreviousAction(env)
wrapped_env = Antialias(wrapped_env)
wrapped_env = DiscreteAction(Flatten(wrapped_env))
# 3. Add Markovian state to the info dict
wrapped_env = Markovian(wrapped_env, Observability.FULL_IN_INFO_DICT)POPGym provides 13 different memory models designed to work with RLlib. You can use these models by specifying them in the custom_model field of your RLlib configuration. You can inspect available hyperparameters using MODEL_CONFIG and BASE_CONFIG on the model class.
When configuring a model like GRU, you can override several architectural components in custom_model_config, including:
preprocessor: An MLP feeding the memory model.preprocessor_input_size and preprocessor_output_size: Dimensions for the preprocessor.hidden_size: The size of the recurrent state.postprocessor: An MLP following the memory model.postprocessor_output_size: Output dimension of the postprocessor.actor and critic: The actor and critic network branches.num_recurrent_layers.Note that when using temporal backpropagation, parameters like max_seq_len, sgd_minibatch_size, train_batch_size, rollout_fragment_length, and horizon should be scaled relative to your Backpropagation Through Time (BPTT) size.
import popgym
import ray
from torch import nn
from popgym.baselines.ray_models.ray_gru import GRU
# Inspect hyperparameters
print(GRU.MODEL_CONFIG)
print(GRU.BASE_CONFIG)
bptt_size = 1024
config = {
"model": {
"max_seq_len": bptt_size,
"custom_model": GRU,
"custom_model_config": {
"preprocessor_input_size": 128,
"preprocessor_output_size": 64,
"preprocessor": nn.Sequential(nn.Linear(128, 64), nn.ReLU()),
"hidden_size": 128,
"postprocessor": nn.Sequential(nn.Linear(128, 64), nn.ReLU()),
"postprocessor_output_size": 64,
"actor": nn.Linear(64, 64),
"critic": nn.Linear(64, 64),
"num_recurrent_layers": 1,
},
},
"sgd_minibatch_size": bptt_size * 4,
"train_batch_size": bptt_size * 8,
"env": "popgym-ConcentrationEasy-v0",
"rollout_fragment_length": bptt_size,
"framework": "torch",
"horizon": bptt_size,
"batch_mode": "complete_episodes",
}
ray.tune.run("PPO", config=config, stop={"timesteps_total": 50_000})If you need to install from the source repository, clone the repository and install the current directory using pip.
git clone https://github.com/smorad/popgym
cd popgym
pip install .You can list all available environment classes in POPGym by accessing the popgym.envs.ALL dictionary. For detailed descriptions of each environment, refer to the popgym.envs module documentation.
import popgym
env_classes = popgym.envs.ALL.keys()
print(env_classes)To install from the source repository including the [baselines] extra, use the following commands:
git clone https://github.com/smorad/popgym
cd popgym
pip install ".[baselines]"To use the standard POPGym environments, install the package using pip. This installs the core environments which require numpy and gymnasium.
pip install popgymImporting the popgym package automatically monkeypatches gym.make to include all POPGym environments. This allows you to instantiate POPGym environments using the standard Gymnasium API, just as you would with any other Gymnasium environment.
import gym
import popgym # This registration happens on import
# Now you can use gym.make to create POPGym environments
env = gym.make('YourPopGymEnvID')POPGym environments follow the Gymnasium interface. You can use wrappers from popgym.wrappers to modify observations and action spaces. Common wrappers include:
PreviousAction: Appends the previous action to the observation.Antialias: Applies antialiasing.Flatten: Flattens observation and action spaces.DiscreteAction: Maps multi-discrete action spaces to a single discrete action (useful for Q-learning).import popgym
from popgym.wrappers import PreviousAction, Antialias, Flatten, DiscreteAction
# Initialize an environment
env = popgym.envs.PositionOnlyCartPoleEasy()
print(env.reset(seed=0))
# Apply wrappers to modify the environment
# This example appends the previous action, flattens the spaces, and discretizes actions
wrapped = DiscreteAction(Flatten(PreviousAction(env)))
print(wrapped.reset(seed=0))When using the Markovian wrapper with Observability.FULL_IN_INFO_DICT, the underlying Markov state is included in the info dictionary returned by step() or reset(). You can access it using the STATE key from popgym.core.observability.
from popgym.core.observability import STATE
# After calling reset() or step()
# obs, reward, terminated, truncated, info = wrapped_env.step(action)
print(info[STATE])