DOCSEARCHbyAlgolia
Submit Docs

skrl Reinforcement Learning Library

repository·develop·Indexed 22 days ago

https://github.com/toni-sm/skrl

A modular and flexible Reinforcement Learning library for Python supporting PyTorch, JAX, and NVIDIA Warp backends. skrl integrates with simulation environments including Isaac Lab, MuJoCo Playground, OpenAI Gym, Gymnasium, PettingZoo, and ManiSkill. It provides implementations for various agents such as PPO, SAC, DDPG, and DQN, and supports simultaneous training by scopes and multi-agent environment wrapping.

Tokens
29.9K
Snippets
76
Records
196
Agent score
73%

What's inside skrl

  1. Overview of skrl utilities and configurations

    develop

    The skrl library provides a comprehensive set of utilities and configurations designed to manage Reinforcement Learning (RL) setups. These tools are designed to work across multiple ML frameworks, including PyTorch, JAX, and NVIDIA Warp.

    Key functional areas include:

    Configurations

    • ML frameworks configuration: Manage settings specific to the underlying ML framework being used.

    Utilities

    • Random seed: Ensure reproducibility across experiments.
    • Spaces: Handle action and observation space definitions.
    • Model instantiators: Simplify the creation of models.
    • Runner: Manage the execution of RL training loops.
    • TensorBoard SummaryWriter: Log training metrics and visualizations.
    • Distributed runs: Support for running experiments in distributed environments.
    • Memory and TensorBoard file post-processing: Tools for processing data after training.
    • Hugging Face integration: Connect RL workflows with the Hugging Face ecosystem.

  2. Overview of SKRL Reinforcement Learning library

    develop

    skrl is an open-source, modular Reinforcement Learning (RL) library written in Python. It is designed for modularity, readability, simplicity, and transparency in algorithm implementation.

    Key features include:

  4. What is Multi-Agent Proximal Policy Optimization (MAPPO)?

    develop

    MAPPO is a model-free, stochastic, on-policy policy gradient algorithm designed for multi-agent environments using CTDE (Centralized Training, Decentralized Execution).

    It utilizes a centralized value function to estimate a single value that guides the policy updates of all agents, which helps improve coordination and cooperation. The algorithm uses Generalized Advantage Estimation (GAE) and a clipped surrogate objective to stabilize training.

  6. What is Independent Proximal Policy Optimization (IPPO)?

    develop

    IPPO is a model-free, stochastic, on-policy policy gradient algorithm designed for multi-agent environments using Decentralized Training, Decentralized Execution (DTDE).

    In IPPO, each agent learns independently using its own local observations of the environment and maintains its own independent critic network to estimate the value function. This makes it suitable for scenarios where agents do not have access to the global state or the actions/observations of other agents during execution.

  8. Overview of skrl Environments

    develop
    In skrl, the environment is the core component where the agent interacts with the world. It is responsible for providing the agent with state information and calculating rewards or penalties based on actions. skrl provides support for loading environments from external frameworks like NVIDIA Isaac Lab and MuJoCo Playground, and offers wrappers to adapt various RL interfaces (Gym, Gymnasium, Isaac Lab, ManiSkill, PettingZoo, Playground, and Shimmy) for use with skrl's single-agent or multi-agent workflows.

  10. How RunningStandardScaler works

    develop

    The RunningStandardScaler implements a running standardization algorithm that updates the mean and variance incrementally as new data arrives. This allows for online feature scaling without needing to see the entire dataset upfront.

    Key Mathematical Operations

    • Standardization: Centering and scaling the input $x$ using the running mean $\bar{x}_t$ and running variance $\sigma^2_t$, with an optional clip_threshold ($c$) and a small epsilon to prevent division by zero: $$\text{clip}((x - \bar{x}_t) / (\sqrt{\sigma^2_t} + \epsilon), -c, c)$$

    • Inverse Transform: Scaling data back to its original representation: $$\sqrt{\sigma^2_t} \cdot \text{clip}(x, -c, c) + \bar{x}_t$$

    • Running Updates: The algorithm uses a parallel variance update method to adjust the running mean ($\bar{x}_t$), running variance ($\sigma^2_t$), and the count of samples ($n_t$) as new data points are processed.

  11. How memories work in skrl

    develop

    Memories are storage components used to collect and reuse experiences. skrl provides generic memory definitions that are not tied to a specific agent implementation.

    • Types: They can serve as replay buffers (for off-policy algorithms like DDPG, TD3, or SAC) or rollout buffers (for on-policy algorithms like PPO or TRPO).
    • Instantiation: Memories are 'empty shells' when instantiated; the agent is responsible for defining the tensors based on its needs.
    • Configuration: Pass memory instances to the agent constructor using the memory or memories argument.

  12. Use RandomMemory for random sampling

    develop

    The RandomMemory class provides a mechanism for random sampling of experiences. It is available for different backends including PyTorch, JAX, and Warp. You can use it to store transitions and sample them randomly during training.

    Note: The specific implementation details (imports and initialization) depend on your chosen backend.

    # Example usage pattern (backend-specific snippets are available in the repository)
# For PyTorch:
from skrl.memories.torch.random import RandomMemory

memory = RandomMemory(capacity=1000)
# ... store and sample transitions ...