Ray

RLlib is an open-source library designed for production-level, highly scalable, and fault-tolerant reinforcement learning (RL) workloads. It provides unified APIs that support a wide variety of industry applications, including:

• Single-agent and Multi-agent training: Training policies for one or many agents simultaneously.
• Offline RL: Training policies from historical datasets.
• External Simulators: Connecting to externally managed simulators for environment interaction.

RLlib is designed to enable developers to move from experimentation to production-scale workloads quickly.

The Ray Distributed Debugger provides a debugger backend and a VS Code extension frontend to streamline debugging distributed Ray applications. It enables two primary workflows:

1. Break into remote tasks: Set breakpoints in any remote task. When a breakpoint is hit, execution pauses, allowing you to connect via VS Code to inspect the state.
2. Post-mortem debugging: When a Ray task fails due to an unhandled exception, Ray automatically freezes the failing task and waits for the debugger to attach, allowing you to inspect the program state at the exact moment of the error.

Note: The frontend is only available in VS Code or VS Code-compatible IDEs like Cursor.

RLlib provides a catalog of built-in reinforcement learning algorithms categorized by their learning paradigm (On-Policy, Off-Policy, Model-based, etc.). Most algorithms support both single-agent and multi-agent setups, as well as multi-GPU training on single nodes or multi-node clusters.

Key categories include:

• On-Policy: e.g., PPO
• Off-Policy: e.g., DQN/Rainbow, SAC
• High-throughput: e.g., APPO, IMPALA
• Model-based RL: e.g., DreamerV3
• Offline RL and Imitation Learning: e.g., BC, CQL, IQL, MARWIL
• Algorithm Extensions: e.g., Curiosity-driven Exploration (ICM)

Soft Actor Critic (SAC) is a state-of-the-art (SOTA) model-free, off-policy Reinforcement Learning (RL) algorithm designed for continuous-control domains.

Key characteristics:

• Maximum-Entropy Framework: Unlike standard RL which only maximizes cumulative reward, SAC optimizes for both the sum of rewards and the expected entropy of the current policy. This helps combat high sample complexity and improves training stability.
• Actor-Critic Framework: It utilizes an actor-critic architecture and includes optimization for the entropy coefficient.
• Discrete Variant: RLlib also implements SAC-Discrete, a variant of the algorithm suitable for discrete action spaces.

Ray provides a suite of specialized libraries for scaling machine learning workloads:

• Data: Scalable Datasets for ML.
• Train: Distributed Training.
• Tune: Scalable Hyperparameter Tuning.
• RLlib: Scalable Reinforcement Learning.
• Serve: Scalable and Programmable Serving.

DreamerV3 is an RLlib-based implementation of the model-based reinforcement learning algorithm by Google DeepMind (2023), implemented in PyTorch.

It works by training a world model in a supervised fashion using real environment interactions. The world model predicts transition dynamics (next state, rewards, and episode continuation). The actor and critic are then trained solely on 'dreamed' trajectories produced by this world model rather than real environment data.

RLlib provides a checkpointing system to save and restore the state of ray.rllib.algorithms.algorithm.Algorithm instances and their subcomponents (such as RLModule). This allows you to:

1. Continue training: Save the state of an experiment and resume it later.
2. Deploy models: Extract specific subcomponents, like a trained RLModule, and deploy them into production without the full RLlib training overhead.

Checkpoints are stored as directories on local disk or in cloud storage locations supported by PyArrow (e.g., GCS or S3).

The tune CLI (Experimental) provides a command-line interface for listing, inspecting, and managing Ray Tune experiments. It allows you to monitor trial progress and export experiment data directly from your terminal.

# Example usage
tune list-trials [EXPERIMENT_DIR]

Ray Serve performance tuning involves understanding the underlying architecture (HTTP proxy actors and deployment replica actors) and applying specific optimizations to the request path. Key areas for tuning include:

• Asynchronous execution: Using async methods to improve concurrency.
• Timeout management: Setting end-to-end request timeouts or overriding timeout and disconnect behavior on a per-request basis.
• Replica selection and routing: Tuning backoff times during replica selection, setting timeouts for queue length probing, and configuring locality-based routing (e.g., enabling same-node routing).
• Throughput optimization: Enabling throughput-optimized serving modes.

The ray-ml image is an extension of the standard rayproject/ray image. It is pre-provisioned with all extended requirements for using RLlib, Serve, and Tune, making it a recommended starting point for exploring the Ray ecosystem.

Note: ray-ml images are not built for the arm64 (aarch64) architecture.

# Example usage (conceptual)
docker pull rayproject/ray-ml:latest