dstack Documentation

dstack is an open-source control plane designed for GPU provisioning and orchestration. It allows you to manage workloads across GPU clouds, Kubernetes, and on-premise clusters using fleets.

Core Components:

1. dstack server: The backend that can run locally, remotely, or via managed dstack Sky.
2. dstack CLI: The primary interface for applying configurations and managing resources (fleets, volumes, gateways, etc.). It uses project configurations stored in ~/.dstack/config.yml.
3. dstack configuration files: YAML files with the .dstack.yml extension used to define workloads.

Common Use Cases:

• Managing dev environments, tasks, or services.
• Creating or applying *.dstack.yml configurations.
• Managing fleets, volumes, and gateways.
• Checking available GPU offers.

dstack is a unified control plane for GPU provisioning and orchestration. It acts as an orchestration layer between AI frameworks/models and the underlying compute resources (GPU clouds, Kubernetes, or on-prem clusters).

Key features:

• Hardware Support: Out-of-the-box support for NVIDIA, AMD, TPU, and Tenstorrent accelerators.
• Compatibility: Works with any hardware, open-source tools, and frameworks (e.g., PyTorch, vLLM, SGLang, Hugging Face).
• Orchestration Capabilities: Manages Fleets, Dev environments, Tasks, Services, and Volumes.
• Automation: Automatically handles infrastructure provisioning, job scheduling, auto-scaling, port-forwarding, and ingress.

Every user is issued a unique token, which can be retrieved from the user account settings page. This token is required for authentication when:

• Logging into the control plane UI.
• Using the dstack CLI.
• Making calls to the dstack HTTP API.

dstack supports POSIX cron syntax for scheduling tasks. Note that in dstack, days of the week start from Monday (0 corresponds to Monday) instead of Sunday.

Fields accept abbreviated English names (e.g., jan–dec, mon–sun).

Cron Field Order:

1. Minute (0-59)
2. Hour (0-23)
3. Day of the month (1-31)
4. Month (1-12 or jan-dec)
5. Day of the week (0-6 or mon-sun)

    ┌───────────── minute (0-59)
    │ ┌───────────── hour (0-23)
    │ │ ┌───────────── day of the month (1-31)
    │ │ │ ┌───────────── month (1-12 or jan-dec)
    │ │ │ │ ┌───────────── day of the week (0-6 or mon-sun)
    │ │ │ │ │
    │ │ │ │ │
    * * * * *

dstack uses custom-built VM images for certain backends to optimize performance. These images are pre-optimized for dstack, such as containing pre-pulled dstack Docker images, which reduces the startup time of dstack jobs compared to standard OS images.

Common image naming conventions include:

• dstack-X.Y: CPU-only instances.
• dstack-cuda-X.Y: NVIDIA GPU instances.
• dstack-grid-X.Y: Specific to Azure for instances requiring NVIDIA Grid drivers.

The cloud GPU market is categorized by target scale and automation maturity. Providers fall into five main archetypes:

• Classical hyperscalers: General-purpose clouds with GPU SKUs (e.g., AWS, Google Cloud, Azure, OCI).
• Massive neoclouds: GPU-first operators with dense clusters (e.g., CoreWeave, Lambda, Nebius, Crusoe).
• Rapidly-catching neoclouds: Smaller, aggressive GPU-first players (e.g., Runpod, DataCrunch, Voltage Park, TensorWave, Hot Aisle).
• Cloud marketplaces: Orchestration and unified API providers that don't own capacity (e.g., NVIDIA DGX Cloud (Lepton), Modal, Lightning AI, dstack Sky).
• DC aggregators: Market-driven aggregators of idle datacenter capacity (e.g., Vast.ai).

You can use the ${{ run.project_name }} variable within the domain field. This ensures that when a gateway is exported to multiple projects, each project receives a unique domain name based on its own project name.

type: gateway
name: global-gateway
backend: aws
region: eu-west-1
domain: ${{ run.project_name }}.mycompany.example

dstack handles three failure types: provisioning failures (no-capacity), job failures (error), and interruptions (interruption).

To enable automatic retries for a task, use the retry property in your .dstack.yml configuration. Note that for distributed tasks, dstack uses an all-or-nothing approach: if any job fails, all jobs are stopped and the entire run is resubmitted.

Important: dstack does not support graceful shutdown signals (unlike Slurm's SIGTERM). Applications must implement proactive checkpointing (periodic saves) and check for existing checkpoints on startup to resume successfully after a retry.

retry:
  on_events: [error, interruption]
  duration: 48h

rest_plugin is a built-in dstack plugin that enables you to implement custom plugin logic by hosting an API server. Instead of writing native dstack plugin code, you can write your logic in any language or framework that can expose a RESTful interface, and dstack will interact with it via HTTP.

The API structure for a rest_plugin server is defined by an OpenAPI specification. You should ensure your API server adheres to this specification to be compatible with dstack.

LLM inference consists of two phases: prefill (processing prompt tokens in parallel, compute-intensive) and decode (generating tokens one by one, memory/bandwidth-intensive).

Prefill–Decode disaggregation separates these phases across dedicated workers to reduce interference and improve hardware efficiency. The prefill–decode ratio defines how many workers are allocated to each phase.

Common workload profiles influence the ideal ratio:

• Prefill-heavy (Input Sequence Length > Output Sequence Length): e.g., summarization.
• Decode-heavy (Input Sequence Length < Output Sequence Length): e.g., reasoning/chain-of-thought.
• Balanced (Input Sequence Length ≈ Output Sequence Length): e.g., translation.