DOCSEARCHbyAlgolia
Submit Docs

ReinforcementLearning.jl

repository·main·Indexed 20 days ago

https://github.com/juliareinforcementlearning/reinforcementlearning.jl

A modular framework for reinforcement learning research in Julia designed for reusability and reproducibility. It provides core abstractions including AbstractPolicy for agent behavior, AbstractEnv for environment management, AbstractTrajectory for experience collection, and AbstractApproximator for value estimation. The ecosystem includes specialized subpackages such as ReinforcementLearningBase.jl, ReinforcementLearningEnvironments.jl, and ReinforcementLearningFarm.jl for tested algorithm implementations.

Tokens
25.8K
Snippets
70
Records
105
Agent score
68%

What's inside ReinforcementLearning.jl

  1. Overview of ReinforcementLearningFarm.jl

    main
    ReinforcementLearningFarm.jl is a collection of updated and tested reinforcement learning algorithms designed specifically for compatibility with ReinforcementLearning.jl version 0.11 and newer. It serves as a stable alternative to ReinforcementLearningZoo, providing 'domesticated' (tested and compatible) implementations of various algorithms.

  2. Overview of Offline Reinforcement Learning algorithms in ReinforcementLearning.jl

    main

    ReinforcementLearning.jl supports a variety of offline reinforcement learning algorithms designed to handle policy constraints. The implemented methods include:

    • Distribution matching
    • Support constraint
    • Implicit constraint
    • Behavior cloning

    These algorithms are designed to work across both discrete and continuous action spaces and have been validated using various dataset types, such as random, medium, and expert datasets.

  3. Implement Multi-Agent Reinforcement Learning Algorithms in Julia

    main

    This technical report details the implementation of several Multi-Agent Reinforcement Learning (MARL) algorithms within the ReinforcementLearning.jl ecosystem, specifically integrated into ReinforcementLearningZoo.jl.

    Key algorithms implemented include:

    • Neural Fictitious Self-play (NFSP)
    • Multi-agent Deep Deterministic Policy Gradient (MADDPG)
    • Exploitability Descent (ED)

    These algorithms are designed to work with various environments, including KuhnPokerEnv and SpeakerListenerEnv, as well as 3rd party environments via OpenSpielEnv.

  4. Overview of ReinforcementLearningBase.jl

    main
    The ReinforcementLearningBase module serves as the foundational layer for the ReinforcementLearning.jl ecosystem. It defines the core abstractions and interfaces that allow different reinforcement learning algorithms, environments, and agent implementations to interoperate. Users typically interact with this base layer indirectly when using higher-level packages, but it is the source of truth for the standard interfaces used across the library.

  5. Implement DQN-style Neural Network Approximators

    main

    When moving from tabular methods to Deep Q-Networks (DQN) to handle large state spaces, two key changes are required in your implementation:

    1. Trajectory Type: Use CircularArraySARTTrajectory instead of VectorSARTTrajectory to facilitate sampling minibatches.
    2. Approximator Type: Replace the TabularApproximator with a NeuralNetworkApproximator.

    For algorithms like Prioritized DQN, you may also need to implement a custom Trajectory that handles priority traces, ensuring transitions are inserted correctly at the PostActStage.

  6. Understand ReinforcementLearningEnvironments.jl traits

    main

    ReinforcementLearningEnvironments.jl uses a set of traits (borrowed from OpenSpiel) to categorize the characteristics of different environments. These traits help developers understand the mathematical and structural properties of an environment, such as whether it is stochastic, multi-agent, or provides partial information.

    Key trait categories include:

    • ActionStyle: MinimalActionSet vs FullActionSet.
    • ChanceStyle: Stochastic, Deterministic, or ExplicitStochastic.
    • DefaultStateStyle / StateStyle: Observation, InformationSet, or InternalState.
    • DynamicStyle: Simultaneous or Sequential dynamics.
    • InformationStyle: PerfectInformation or ImperfectInformation.
    • NumAgentStyle: SingleAgent or MultiAgent.
    • RewardStyle: TerminalReward or StepReward.
    • UtilityStyle: GeneralSum, ZeroSum, ConstantSum, or IdenticalUtility.

  7. How reinforcement learning components work together

    main

    The ReinforcementLearning.jl framework uses a modular design where different aspects of an experiment are decoupled into specific abstractions. This allows users to easily swap out algorithms, environments, or monitoring tools.

    • AbstractPolicy: Defines the agent's behavior. A policy takes the current state of the environment and returns an action.
    • AbstractEnv: Defines the world the agent interacts with. It manages states, transitions, and rewards.
    • Stop Condition: Controls the lifecycle of the experiment, determining when to stop based on steps, episodes, or other criteria.
    • AbstractHook: Provides a way to observe the experiment without modifying the core logic. Hooks are used for logging, data collection, and visualization.

  8. Configure ReinforcementLearningTrajectories

    main

    Trajectories are managed via the ReinforcementLearningTrajectories.jl package and consist of three components:

    1. Container (AbstractTraces)

    Stores the actual data. Common types include:

    • CircularArraySARTSTraces: A fixed-length container for :state, :action, :reward, and :terminal (aliased as SART).
    • MultiplexTraces: Stores two names in one container (e.g., (:state, :next_state)). When sampling :next_state at index i, it returns the value at i+1.
    • Traces: Simple named containers for single values like reward or terminal status.
    • CircularArrayBuffer: Preallocated arrays that overwrite the oldest elements when full. Dimensions are passed as a tuple, with the last dimension being the capacity.

    2. Controller

    Decides when the trajectory is ready to be sampled.

    • InsertSampleRatioController(ratio, threshold): Samples a batch when the ratio of samples / insertions exceeds the specified ratio. The threshold defines the minimum number of insertions required before sampling begins.

    3. Sampler

    Fetches data to create batches.

    • BatchSampler{names}(batchsize, rng): Samples batchsize elements. The names argument specifies which traces to query (e.g., (:state, :action, :next_state)).

  9. Handle DefaultStateStyle for algorithm compatibility

    main

    Algorithm developers typically call state(env) without specifying a style. This call is dispatched to state(DefaultStateStyle(env), env).

    If your environment supports multiple state representations, you must define a DefaultStateStyle. You can use the DefaultStateStyleEnv wrapper to override the pre-defined default behavior for your environment.

  10. Use BEAR (Bootstrapping Error Accumulation Reduction)

    main

    BEAR is a policy-constraint offline RL method that uses support constraint (via Maximum Mean Discrepancy - MMD) instead of distribution matching. It trains a VAE to simulate sampling actions from the dataset, and the Actor's loss is regularized by the MMD between the sampled actions and the actor's actions.

    Key parameters for BEARLearner:

    • ε: Used to update the Lagrangian multiplier.
    • p: Hyper-parameter for state repetition.
    • max_log_α / min_log_α: Clamps for the Lagrangian multiplier log_α.
    • sample_num: Number of samples used to calculate MMD loss.
    • mmd_σ: Adjusts the size of the MMD loss.
    • kernel_type: Specifies the MMD calculation method, either :laplacian or :gaussian.

    Note: This algorithm does not support GPU acceleration.

    mutable struct BEARLearner{BA1, BA2, BC1, BC2, V, L} <: AbstractLearner
    policy::BA1
    target_policy::BA2
    qnetwork1::BC1
    qnetwork2::BC2
    target_qnetwork1::BC1
    target_qnetwork2::BC2
    vae::V
    log_α::L
    ε::Float32
    p::Int
    max_log_α::Float32
    min_log_α::Float32
    sample_num::Int
    kernel_type::Symbol
    mmd_σ::Float32
end