SkyPilot Documentation

SkyPilot uses a YAML interface to specify clusters, jobs, or services. You can define resource requirements, setup commands, run commands, file mounts, and storage mounts.

Key characteristics:

• All fields are optional: You only need to specify the fields relevant to your task; otherwise, defaults are used.
• Usage: YAML files can be used via the SkyPilot CLI or the programmatic API (e.g., using sky.Task.from_yaml()).

SkyServe is SkyPilot's model serving library (currently in beta). It allows you to deploy existing serving frameworks (such as vLLM, TGI, or FastAPI) across multiple regions or clouds.

Key features include:

• Multi-cloud/Multi-region deployment: Scale replicas across different locations to reduce costs and increase availability.
• Automatic Load Balancing: A centralized controller distributes traffic to replicas using a least-load algorithm.
• Autoscaling and Recovery: Handles replica scaling and automatically re-launches failed replicas.
• Privacy: Everything runs inside your own cloud accounts and VPCs.

Note: For production-ready requirements (high performance, multi-cluster design, built-in observability, and advanced security), consider SkyPilot Endpoints.

Sky Serve is a serving library for SkyPilot designed to expose a single endpoint that distributes incoming traffic across multiple serving endpoints. These endpoints can reside on different resources, regions, and clouds. Sky Serve provides transparent handling for:

• Load balancing: Distributing traffic across healthy endpoints.
• Failover: Managing requests when endpoints become unhealthy.
• Autoscaling: Adjusting the number of serving endpoints based on demand.

SkyPilot provides three core abstractions depending on your workflow stage:

1. SkyPilot Clusters (sky launch / sky exec): Best for interactive development and debugging. Use these to launch a cluster, SSH into it, or connect VSCode/Cursor to iterate quickly. The cluster stays up until manually stopped or autostop triggers.
2. Managed Jobs (sky jobs launch): Best for long-running training and batch jobs. These run unattended and manage the full lifecycle, including automatic recovery from spot preemptions, quota limits, and transient failures.
3. SkyServe (sky serve up): Best for production model serving. Provides autoscaling, load balancing, and multi-cloud replicas for model serving endpoints and API services.

SkyServe uses a centralized controller VM to manage deployments. The architecture consists of:

1. Controller: Monitors replica status, re-launches failed replicas, and handles autoscaling. It is launched in the cloud with the best price/performance ratio.
2. Load Balancer: A lightweight HTTP server that listens on the service endpoint and distributes incoming traffic to ready replicas.

All process groups share a single controller VM, which can be customized for specific resource requirements.

SkyPilot provides basic Role-Based Access Control (RBAC) with two supported roles:

• User: Can launch and manage resources such as clusters and jobs.
• Admin: Can manage SkyPilot API server settings, users, and workspaces.

Note: RBAC support is only available when using SSO authentication. It is not supported when using basic authentication.

SkyPilot distinguishes between two types of cluster names to ensure compatibility with cloud provider constraints and to prevent naming collisions between users in the same organization:

1. SkyPilot cluster name (cluster_name): The original "display" name you choose or that the system generates. This is the name you use in CLI commands and APIs (e.g., sky launch -c <cluster_name>).
2. Cluster name on cloud (cluster_name_on_cloud): An auto-generated name used when communicating with cloud providers (e.g., by ray up or node_provider). This name is automatically truncated to meet cloud length limits and appended with a user hash to ensure uniqueness across different users.

Note that the cloud name is also recorded in the VM tag ray-cluster-name.