DOCSEARCHbyAlgolia
Submit Docs

Ray Documentation

website·Indexed 40 days ago

https://docs.ray.io/en/latest/

Documentation for Ray, featuring Ray Data for distributed data processing, Ray Train for scalable model training, and Ray Tune for hyperparameter tuning. Includes benchmarks for image classification and multimodal inference, guides on avoiding out-of-memory (OOM) errors, and tutorials for integrating PyTorch with Ray Tune.

Tokens
469.3K
Snippets
1.9K
Records
3.1K
Agent score
74%

What's inside Ray

  1. Overview of Ray Serve LLM observability and monitoring

    Ray Serve LLM provides built-in observability features to monitor LLM deployments, including service-level metrics for request and token behavior, vLLM-specific engine metrics (such as Time to First Token (TTFT) and Time Per Output Token (TPOT)), and Prometheus integration for exporting metrics to custom monitoring and alerting systems. Pre-built Grafana dashboards are also available for LLM-specific visualizations.

  2. Overview of Ray Sandboxes with KubeRay

    Ray Sandboxes (ray.experimental.sandbox) allow for the safe execution of untrusted, model-generated code (such as RL rollout workers or LLM agents) within lightweight, kernel-isolated environments. By running gVisor (runsc) directly inside Ray worker Pods, the system achieves sub-100 ms startup latencies and high density, allowing hundreds of concurrent sandboxes per node without the overhead of provisioning individual Kubernetes Pods. This is compatible with Google Kubernetes Engine (GKE) and other Kubernetes distributions.

  3. Overview of Ray Tune data and workload management

    Ray Tune provides a unified abstraction for managing ML training loops across various hardware configurations (CPU, GPU, and multi-node clusters). Key capabilities include fault tolerance via checkpointing to resume from failures or pre-emptions, seamless scalability from local prototyping to cluster-scale training, and observability through integrated metrics, logs, and checkpoints via Ray and Anyscale tooling.

  4. Overview of Ray Serve LLM deployment

    Ray Serve LLM is designed for deploying large language models in production. It leverages Ray Serve primitives to support distributed, multi-node serving and provides an OpenAI-compatible API. Key capabilities include support for multi-model serving, tensor and pipeline parallelism, LoRA, and backends such as vLLM and SGLang.

  5. Overview of KAI Scheduler for AI/ML workloads

    KAI Scheduler is a scalable Kubernetes scheduler designed for orchestrating GPU clusters at scale. It supports the full AI lifecycle, from interactive development to distributed training and inference. Key capabilities include:

    • Bin packing and spread scheduling: Optimizes node usage by either minimizing fragmentation or increasing resiliency/load balancing.
    • GPU sharing: Consolidates multiple Ray workloads from different teams on the same GPU to reduce idle time.
    • Workload and Cluster autoscaling: Scales Ray replicas/workers within constraints and is compatible with cloud auto-scalers like Karpenter.
    • Workload priorities: Manages Ray workload priority within queues.
    • Hierarchical queues and fairness: Implements two-level queues with quotas, limits, and equitable resource distribution using Dominant Resource Fairness (DRF).

  6. Overview of KubeRay integration with Volcano

    KubeRay integrates natively with Volcano, a batch scheduling system for Kubernetes. This integration allows RayCluster, RayJob, and RayService to utilize Volcano's advanced scheduling capabilities, such as gang scheduling, job queues, fair scheduling policies, and network topology-aware scheduling. This is particularly useful for improving the efficiency of Ray pod scheduling in multi-tenant Kubernetes environments.

  7. Overview of Ray Train for Distributed Training

    Ray Train is a distributed training framework designed to scale PyTorch, XGBoost, and LightGBM workloads across multiple GPUs. It abstracts the complexity of managing Distributed Data Parallel (DDP) by handling infrastructure orchestration, autoscaling, and data sharding. Key benefits include reduced setup overhead, built-in observability via the Ray dashboard, and fault tolerance through automatic retries and checkpointing.

  8. Overview of the Experimental Ray Serve Java API

    Ray Serve provides a native Java API that allows developers to create, query, update, and manage deployments. The API supports cross-language interoperability, meaning Java deployments can be called via Python and Python deployments can be managed using the Java API. Note that this feature is currently experimental and subject to change, and it is not supported on KubeRay.

  9. Overview of launching Ray clusters on Cloud VMs

    Ray provides built-in support for launching clusters on AWS, GCP, and Azure, with community-maintained integrations for Aliyun and vSphere. A Ray cluster consists of one head node and multiple worker nodes. It supports heterogeneous compute nodes, including GPU nodes, and optional autoscaling to dynamically add or remove worker nodes based on workload requirements.

  10. Overview of Asynchronous Proximal Policy Optimization (APPO)

    APPO (originally known as IMPACT) is an asynchronous variant of PPO based on the IMPALA architecture. It uses a surrogate policy loss with clipping and allows multiple SGD passes per collected train batch. It utilizes v-trace to account for off-policy data since weights are not always synced back to EnvRunners immediately. APPO scales using multiple EnvRunners and multiple GPU- or CPU-based Learners. Configuration is handled via ray.rllib.algorithms.appo.appo.APPOConfig.

  11. Overview of Ray Serve for model serving

    Ray Serve is a scalable, framework-agnostic model serving library used to build online inference APIs. It supports a wide range of models, including PyTorch, TensorFlow, Keras, Scikit-Learn, and general Python business logic. Key capabilities include:

    • LLM Optimizations: Response streaming, dynamic request batching, and multi-node/multi-GPU serving.
    • Model Composition: Ability to build complex inference pipelines consisting of multiple models and logic in Python.
    • Resource Efficiency: Built on Ray, allowing for flexible scheduling and fractional GPU allocation to serve multiple models cost-effectively.