DOCSEARCHbyAlgolia
Submit Docs

Gymnasium

repository·main·Indexed 11 days ago

https://github.com/farama-foundation/gymnasium

An open-source Python library providing a standardized API for communication between reinforcement learning algorithms and environments. As the successor to OpenAI's Gym, it includes a diverse set of reference environments across families such as Classic Control, Box2D, Toy Text, MuJoCo, and Atari. It features the gymnasium.Env base class for custom environments, a registry system for environment management, and support for functional environments via FuncEnv and FunctionalJaxEnv.

Tokens
39.8K
Snippets
108
Records
240
Agent score
95%

What's inside Gymnasium

  1. Overview of MuJoCo environments

    main

    MuJoCo (Multi-Joint dynamics with Contact) environments in Gymnasium are physics-based simulations for robotics and biomechanics. They are generally more difficult to solve via policy than other Gymnasium environments.

    Available Environments

    RobotShort Description
    CartPoles
    InvertedPendulumMuJoCo version of CartPole (Continuous actions)
    InvertedDoublePendulum2 Pole variation of CartPole
    Arms
    Reacher2d arm reaching for an object
    Pusher3d arm pushing an object to a target
    2D Runners
    HalfCheetah2d quadruped running
    Hopper2d monoped hopping
    Walker2d2d biped walking
    Swimmers
    Swimmer3d robot swimming
    Quadruped
    Ant3d quadruped running
    Humanoid Bipeds
    Humanoid3d humanoid running
    HumanoidStandup3d humanoid standing up

    State Space Composition

    The state space consists of two parts concatenated together:

    1. Body part and joint positions (mujoco.MjData.qpos)
    2. Corresponding velocities (mujoco.MjData.qvel)

    All environments are stochastic, using Gaussian noise added to a fixed initial state.

  2. Overview of Box2D environments

    main

    Box2D environments in Gymnasium are physics-based toy games that use the Box2D physics engine and PyGame-based rendering. These environments are widely used as benchmarks for reinforcement learning.

    Common environments in this category include:

    • BipedalWalker
    • CarRacing
    • LunarLander

    Note that these environments are highly configurable via arguments; check the specific documentation for each environment to see available parameters.

  3. Overview of Gymnasium environment families

    main

    Gymnasium provides several categories of environments:

    • Classic Control: Physics-based problems (e.g., CartPole).
    • Box2D: Toy games using Box2D physics and PyGame rendering.
    • Toy Text: Extremely simple environments with small discrete spaces, ideal for debugging.
    • MuJoCo: Complex multi-joint physics control environments.
    • Atari: Emulated Atari 2600 ROMs.
    • Third-party: External environments compatible with the Gymnasium API. When using these, you may need to use apply_env_compatibility in gym.make if the environment was built for a different version.

  4. Overview of Classic Control environments

    main

    The Classic Control suite includes five environments:

    • Acrobot (includes noise applied to actions)
    • CartPole
    • Mountain Car
    • Mountain Car Continuous
    • Pendulum

    Key characteristics:

    • Stochasticity: All environments have stochastic initial states within a given range.
    • Difficulty: These are generally considered easier environments to solve with a policy compared to other Gymnasium environments.
    • Configurability: Each environment is highly configurable via arguments passed during instantiation.

  5. Explore Farama Foundation first-party environments

    main

    The Farama Foundation maintains several specialized environment libraries that are compatible with the Gymnasium API. These include environments for robotics, gridworlds, 3D navigation, web interaction, and more.

    Key first-party environment projects include:

  7. Composite Spaces in Gymnasium

    main

    Composite spaces allow you to combine fundamental spaces to represent complex data structures, which is essential for vectorized environments or multi-agent setups.

    • Dict: A dictionary of keys mapping to subspaces. Use this for unordered collections of different data types.
    • Tuple: A tuple of subspaces. Use this for ordered collections of different data types.
    • Sequence: A variable number of instances of a single subspace. Useful for entities or variable-length action sets.
    • Graph: Supports graph-based observations or actions with discrete or continuous nodes and edge values.
    • OneOf: Supports optional action spaces where an action can be one of $N$ possible subspaces.

  8. Using third-party environments built for Gym

    main

    Many existing reinforcement learning environments were originally built using the OpenAI gym library. While many of these can be adapted to work with gymnasium, their functionality is not guaranteed.

    To use these environments, you should refer to the Compatibility with Gym guide to understand the necessary migration steps or wrappers required to make them compatible with the gymnasium API.

  9. Modify environments using Wrappers

    main

    Wrappers are used to modify an existing environment's behavior or observation/action spaces without altering the underlying environment code. They act as filters or modifiers and can be chained together.

    Most environments created via gymnasium.make are automatically wrapped with TimeLimit, OrderEnforcing, and PassiveEnvChecker by default.

    To use a wrapper, initialize your base environment and pass it as the first argument to the wrapper's constructor.

    import gymnasium as gym
from gymnasium.wrappers import FlattenObservation

# Start with a complex observation space
env = gym.make("CarRacing-v3")
print(env.observation_space.shape)  # (96, 96, 3)

# Wrap it to flatten the observation into a 1D array
wrapped_env = FlattenObservation(env)
print(wrapped_env.observation_space.shape)  # (27648,)

  10. Understand the terminated and truncated distinction

    main

    In Gymnasium (v0.26+), the single done flag from OpenAI Gym v0.21 has been split into two distinct boolean flags: terminated and truncated. This distinction is critical for Reinforcement Learning (RL) algorithms, particularly for correct value function bootstrapping.

    • terminated: The episode ended because the task was completed or failed (e.g., the agent reached a goal, died, or crashed). In this case, there is no future value: next_value = 0.
    • truncated: The episode ended due to an external constraint, such as a time limit or a maximum step limit. In this case, the agent might have been in a good state, so you should bootstrap the value: next_value = value_function(next_obs).

    Implementation Strategy

    Simple migration (for basic loops):

    obs, reward, terminated, truncated, info = env.step(action)
done = terminated or truncated

    Correct RL bootstrapping:

    obs, reward, terminated, truncated, info = env.step(action)
if terminated:
    # Natural ending - no future value
    target = reward
elif truncated:
    # Time limit - estimate future value
    target = reward + discount * estimate_value(obs)

  11. Understand the Agent-Environment Loop

    main

    Reinforcement learning in Gymnasium follows a cyclic process known as the agent-environment loop:

    1. Agent observes: The agent receives an observation from the environment (e.g., sensor data or screen pixels).
    2. Agent chooses an action: The agent selects an action based on its policy.
    3. Environment responds: The agent executes the action via env.step(action), and the environment returns a new observation, a reward, and status flags (terminated and truncated).
    4. Repeat: This continues until the episode ends.

    A single exchange of action and observation is called a timestep.

  12. Understand Gymnasium environment versioning

    main
    Gymnasium uses strict versioning for reproducibility. All environment IDs include a version suffix (e.g., -v0, -v1). If changes are made to an environment that could impact learning results, the version number is incremented to ensure that existing research or code remains reproducible with the original behavior.