Llumnix Documentation

Prefill-decode (P-D) disaggregation is an experimental technique where the prefill phase (initial prompt processing) and the decode phase (token-by-token generation) are computed on separate instances.

How it works:

1. Prefill Phase: Requests are dispatched to dedicated prefill instances.
2. Migration: Once the prefill phase is complete, the system migrates the generated Key-Value (KV) cache from the prefill instance to a decode instance.
3. Decode Phase: The decode instance continues the computation.

Benefits:

• Reduced Interference: Separates the two phases to optimize resource utilization.
• Engine Agnostic: Inference engines (like vLLM) do not need to be aware of the P-D distinction; Llumnix handles the scheduling and migration logic.
• Native Integration: Leverages Llumnix's built-in KV cache transfer and decoupled API server mechanisms.

Llumnix is a cross-instance request scheduling layer built on top of LLM inference engines like vLLM. It is designed to optimize multi-instance serving performance through:

• Low Latency: Reduces Time-to-First-Token (TTFT) and queuing delays via KV-cache-aware scheduling, and reduces Time-Between-Tokens (TBT) and preemption stalls via better load balancing.
• High Throughput: Integrates with state-of-the-art engines and supports techniques like prefill-decode disaggregation.

It achieves this using dynamic, fine-grained, KV-cache-aware scheduling and a near-zero overhead KV cache migration mechanism that enables continuous rescheduling across instances for load balancing, de-fragmentation, and disaggregation.

Llumnix has two architectural versions:

• Llumnix v1 (New Repository): A refactored, modular, and cloud-native architecture designed for production environments.
• Llumnix v0 (This Repository): The Ray-based architecture. This version is recommended for local deployments, quick prototyping, and experimentation with scheduling ideas.

Llumnix acts as a request scheduling layer on top of LLM backend engines. Because it is currently built on top of vLLM (version 0.6.3.post1), it supports any model compatible with that version of vLLM.

Primary tested models include:

• Llama
• Llama2
• Llama3
• Qwen
• Qwen1.5
• Qwen2

For a complete list of models that are likely to work, refer to the vLLM Supported Models documentation.

Llumnix currently has limited support for Ray head fault-tolerance.

• Impact on running services: If the Ray head fails, existing Llumnix actors that are already alive will continue to run, meaning current service availability is not immediately impacted.
• Impact on scaling/recovery: During a Ray head failure, Llumnix cannot create or restart any new actors.

Future versions aim to incorporate Ray's high-availability mechanisms to enable full Ray failover.

Llumnix supports disaggregating prefill and decode workloads to improve efficiency.

Disaggregation Modes

• --enable-pd-disagg: Enables prefill-decode disaggregation for vLLM.
• --enable-vllm-v1-engine-pd-disagg: Enables engine-based PD disaggregation for vLLM-V1.
• --enable-bladellm-engine-pd-disagg: Enables engine-based PD disaggregation for BladeLLM.
• --enable-adaptive-pd: [Experimental] Enables adaptive prefill-decode disaggregation.

Configuration

• --pd-ratio: The P:D ratio used in global launch mode (e.g., 1:1).
• --instance-type: Defines the role of the engine instance. Choices: prefill, decode, neutral.

When deploying Llumnix, you can control how the Ray cluster is managed using the --launch-ray-cluster flag:

• Automatic Creation: Including --launch-ray-cluster tells Llumnix to automatically build a Ray cluster during deployment. Warning: This will overwrite any existing Ray cluster.
• Existing Cluster: To run Llumnix components within a pre-existing Ray cluster without overwriting it, simply omit the --launch-ray-cluster option.

Internal Llumnix components are launched as Ray actors. If an actor fails, Ray automatically restarts it. The impact of these failures depends on the specific component:

• API server: If an API server fails, all requests associated with it are aborted. This includes requests that might be running on other instances not co-located with the failed server.
• Scheduler: If the scheduler fails, Llumnix enters a scheduler-bypassing mode. In this mode, API servers use simplified internal scheduling logic to maintain service.
• Instance (llumlet / backend): If an instance (such as a llumlet or a backend engine) fails, all requests currently running on that instance are aborted. This also affects requests dispatched from other API servers that were targeting that specific instance.

Requirements

Llumnix requires Python 3.9~3.10 and is built on top of vLLM (version 0.6.3.post1).

Install from PyPI

You can install the package directly using pip:

pip install llumnix

Build from Source

To build and install from the repository:

git clone https://github.com/AlibabaPAI/llumnix.git
cd llumnix
make vllm_install

Migration Backend Installation

• rayrpc (Default): No additional steps required.
• NCCL: Run make cupy-cuda to install cupy-cuda (ensure it matches your CUDA version).
• Gloo:
  1. Install Bazel >= 5.1.0.
  2. Install cupy-cuda.
  3. Run make pygloo to install pygloo.

Note: Using conda is not recommended as it may fail to handle pygloo's dependency on gcc libstdc++.so.6: version GLIBCXX_3.4.30.