DOCSEARCHbyAlgolia
Submit Docs

Volcano Documentation

repository·master·Indexed 26 days ago

https://github.com/volcano-sh/volcano

Volcano is a Kubernetes-native batch scheduling system designed to optimize high-performance, elastic, and batch workloads such as AI/ML, Big Data, and HPC. The documentation includes a comprehensive benchmark framework for testing gang scheduling and pod scheduling scenarios, featuring support for simulated nodes via KWOK, topology-aware scheduling with HyperNodes, and microsecond-precision latency measurement using an audit-exporter.

Tokens
146.8K
Snippets
281
Records
629
Agent score
91%

What's inside Volcano

  1. Overview of Volcano Batch Scheduling System

    master

    Volcano is a Kubernetes-native batch scheduling system designed to extend and enhance the standard kube-scheduler. It is specifically optimized for managing and optimizing batch and elastic workloads, including:

    • AI/ML/DL: Artificial Intelligence, Machine Learning, and Deep Learning.
    • Big Data: Frameworks like Spark, Flink, and Ray.
    • HPC/Bioinformatics: High-Performance Computing and genomics applications.

    Volcano provides robust integration with various frameworks such as TensorFlow, PyTorch, Argo, MPI, Horovod, and more. It is an incubating project of the Cloud Native Computing Foundation (CNCF).

  3. Overview of Usage Based Scheduling

    master
    Usage based scheduling in Volcano aims to balance the actual resource usage across compute nodes. Unlike standard scheduling which relies on resource requests and node allocatable capacity, usage based scheduling considers the real-time node usage to prevent pods from being scheduled onto nodes that already have high actual utilization but low allocation rates.

  4. Overview of Volcano Batch System

    master

    Volcano is a batch system built on Kubernetes designed to provide advanced scheduling mechanisms for batch and elastic workloads. It is specifically optimized for applications such as:

    1. Machine learning and deep learning
    2. Bioinformatics and genomics
    3. Big data applications

    Volcano integrates with generalized domain frameworks including TensorFlow, Spark, Ray, PyTorch, and MPI.

  5. Understand Volcano Scheduler Tiers and Plugins

    master

    Volcano's scheduling logic is implemented through Plugins. A plugin registers a series of functions that are executed during specific scheduling actions.

    Plugin Structure

    Most plugins consist of three core functions:

    • Name: Provides the unique identifier for the plugin.
    • OnSessionOpen: Executes operations when a scheduling session starts and registers specific scheduling logic functions.
    • OnSessionClose: Cleans up resources when a session finishes.

    Key Considerations

    • Customization: Some plugins accept arguments to match specific user scenarios.
    • Interoperability: Different plugins may register the same functions (e.g., preemptableFn) with different logic. When configuring multiple plugins, ensure they are compatible and can work together.
    • Built-in Plugins: Volcano includes 15 built-in plugins (as of April 2022) covering various scheduling strategies like binpacking, gang scheduling, and DRF.

  6. Planned features for HyperJob: Automatic Job Splitting and Fault Tolerance

    master

    Future updates to HyperJob are planned to include:

    • Automatic Job Splitting: Implementation of SplitPolicy to split jobs across clusters based on accelerator requirements:
      • Static mode: Splits jobs into equal parts based on a specified accelerator count.
      • Auto mode: Integrates with external services to dynamically determine the optimal split based on cluster resource availability.
    • Advanced Fault Tolerance: Automatic job migration and recovery strategies for sub-jobs that fail across clusters.
    • HyperJob Plugins: Framework-specific plugins for multi-cluster coordination.

  7. Understand the Agent Scheduler Architecture

    master

    The Agent Scheduler is a specialized, high-performance scheduler designed for latency-sensitive AI Agent workloads that require ultra-fast task creation and high throughput. Unlike the standard Volcano scheduler which processes workloads in batches, the Agent Scheduler uses an independent 'fast-path' to handle frequent task creation and can utilize different scheduling strategies (e.g., prioritizing resource fragments over topology spread).

    Key components include:

    • Sharding Controller: Dynamically divides cluster nodes into shards based on policies (resource thresholds, node types, etc.) to allow parallel scheduling.
    • Agent Fast-Path Scheduler: Performs rapid scheduling within assigned shards using multiple concurrent workers.
    • Volcano Scheduler: Collaborates with the Agent scheduler by using a sharding coordinator to synchronize nodes, allowing both schedulers to operate on different shards in parallel.

  8. Understand Dominant Resource Fairness (DRF) in Volcano

    master

    Volcano implements Dominant Resource Fairness (DRF) to handle resource allocation across multiple resource types (e.g., CPU, Memory, GPU).

    DRF works by identifying the dominant resource for each job—the resource type that requires the largest share of the total cluster capacity. The scheduler calculates the dominant share (the highest ratio of allocated resource / total resource among all requested types) and aims to maximize the smallest dominant share in the system by prioritizing jobs with the lowest shares.

  9. Understand the Sharding Controller Roadmap

    master

    The Sharding Controller is designed to partition Kubernetes clusters between different scheduler types (e.g., traditional batch vs. Agentic AI workloads) based on resource utilization and node warmup status.

    Planned improvements include:

    • Short-Term: Adding memory utilization to shard strategies, supporting custom node metrics, and enabling dynamic configuration updates without restarts.
    • Medium-Term: Machine learning for predicting node utilization, multi-dimensional scheduling (GPU, network), and hierarchical sharding for large clusters.
    • Long-Term: Cross-cluster sharding, workload-aware sharding, and integration with the Kubernetes Scheduling Framework.

  10. Orchestrate multiple VCJobs using JobFlow

    master

    JobFlow is a specialized orchestration mechanism designed to manage inter-job dependencies between multiple VCJobs. It allows users to declare complex batch job workflows using two primary concepts: JobTemplate (for defining reusable job definitions) and JobFlow (for defining the orchestration logic).

    JobFlow supports complex controlling primitives such as:

    • Sequential execution
    • Parallel execution
    • If-then-else statements
    • Switch-case statements
    • Loop execution

    Unlike general-purpose workflow engines, JobFlow is purpose-built for batch workloads (AI, BigData, HPC) and understands VCJob specific details, such as running states, timestamps, and pod-failure-ratios.

  11. Understand JobFlow Orchestration

    master

    JobFlow (jf) defines the running flow and dependencies of a set of Volcano jobs (vcjobs). It allows for job-dependent operations where jobs are issued based on the completion or status of preceding jobs in a flow.

    Submission Process

    1. Creation: kubectl creates JobTemplate and JobFlow objects.
    2. Orchestration: The JobFlowController uses the JobTemplate to create VcJob objects based on dependency rules.
    3. Scheduling: VcJobController creates Pods/PodGroups, and the vc-scheduler assigns them to nodes.

    Lifecycle Constraints

    • Updates: Updating a JobFlow is currently blocked by a webhook.
    • Deletion: Deleting a JobFlow that is not in a 'complete' state is intercepted by a webhook. If the JobFlow is complete, deleting it will automatically delete all vcjobs created by that flow.

  12. Understand Volume Binding Design in Volcano

    master
    Volcano's volume binding mechanism is designed to support WaitForFirstConsumer mode. Unlike previous implementations that intrusively modified the scheduler cache, the VolumeBinding plugin is integrated into the predicates plugin. This allows volume binding to function as a standard plugin within the scheduler lifecycle without exposing binder interfaces directly to the cache.