DOCSEARCHbyAlgolia
Submit Docs

vLLM Production Stack

repository·main·Indexed 25 days ago

https://github.com/vllm-project/production-stack

A production-ready deployment stack for vLLM that automates GPU-accelerated infrastructure and deployment via Helm. It supports multiple cloud providers including Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), and Oracle Cloud Infrastructure (OCI). The project also includes a multi-round QA benchmarking tool to analyze throughput, latency, and Time to First Token (TTFT) using dummy context or ShareGPT datasets.

Tokens
103.1K
Snippets
246
Records
431
Agent score
81%

What's inside vllm-project/production-stack

  1. Overview of the vLLM Production Stack on Nebius

    main

    The vLLM Production Stack is an opinionated, GPU-first blueprint designed for production vLLM inference on Nebius MK8s (Managed Kubernetes).

    Key Characteristics:

    • Infrastructure: Deployed via Terraform and Helm.
    • Networking: Uses Nebius VPC-Cilium (overlay mode) with VXLAN/Geneve encapsulation and eBPF datapath.
    • Inference Hardware: Supports GPU-only workloads with pre-baked NVIDIA drivers for L40S, H100, H200, or B200 GPUs.
    • Observability: Includes kube-prometheus-stack (Grafana + Prometheus) for GPU and vLLM dashboards.
    • Security: Uses cert-manager with Let's Encrypt (HTTP-01 solver) for SSL/TLS via NGINX Ingress.

  2. Overview of the vLLM Production Stack

    main

    The vLLM Production Stack is a K8S-native, cluster-wide deployment reference implementation designed to build scalable inference stacks on top of vLLM. It enables developers to:

    • Scale seamlessly: Transition from a single vLLM instance to a distributed deployment without modifying application code.
    • Monitor performance: Track metrics via a web dashboard.
    • Optimize throughput: Leverage performance benefits from request routing and KV cache offloading.
    • Multi-cloud deployment: Deploy the stack on AWS, GCP, or other cloud providers.

  3. Choose a vLLM Production Stack deployment option

    main

    The vLLM Production Stack offers three distinct deployment methods depending on your infrastructure requirements and management preference:

    1. Helm Chart Deployment: The standard method for Kubernetes. It uses Helm charts to provide a streamlined deployment process with configurable parameters for models, resource allocation, and routing logic.
    2. Custom Resource Definitions (CRD): Best for advanced configurations and operator-based management. This method integrates deeply with Kubernetes-native workflows and offers maximum flexibility.
    3. Gateway Inference Extension: An advanced option for high-scale environments. It enables inference through gateway infrastructure and supports both individual models and inference pools with sophisticated routing capabilities.

  4. Project Structure for CoreWeave vLLM Deployment

    main

    The repository is organized into Terraform files for infrastructure and a config/ directory for Helm templates and Kubernetes manifests:

    • main.tf: CKS Cluster & NodePool logic.
    • network.tf: CoreWeave VPC & IP Prefixes (Pod/Svc/LB CIDRs).
    • vllm-production-stack.tf: vLLM Helm release logic.
    • cluster-tools.tf: Cert-manager, Traefik, Monitoring, and Metrics-server.
    • provider.tf: CoreWeave, Helm, & Kubectl provider configuration.
    • variables.tf: Configurable knobs and defaults.
    • output.tf: Unified Stack Dashboard output.
    • config/llm-stack/helm/gpu/: Contains vLLM deployment templates (e.g., gpu-gpt-oss-20-cw.tpl).
    • config/manifests/: Kubernetes manifests for Let's Encrypt and NodePools.

  5. Available vLLM Production Stack Tutorials

    main

    The following tutorial modules are available to guide your deployment and configuration:

    1. Install Kubernetes Environment: Setting up the foundational Kubernetes environment.
    2. Minimal Helm Installation: Step-by-step deployment using Helm with minimal configuration.
    3. Basic vLLM Configuration: Customizing vLLM-specific options within the stack.
    4. Load Model from Persistent Volume: Efficiently loading models from PVs to optimize resource usage.
    5. Launch Multiple Models: Deploying and managing multiple models simultaneously.
    6. Offload KV Cache: Offloading the KV cache to CPU to improve production performance.

  6. What is KV cache aware routing

    main

    KV cache aware routing is a feature in the vLLM Production Stack that routes incoming requests to the specific instance with the highest KV cache hit rate.

    Key differences:

    • KV cache aware routing: Prioritizes actual cache hits to optimize resource usage and maximize efficiency. It routes requests based on where the cache is currently available.
    • Prefix aware routing: Always sends requests with the same prefix to the same instance, even if the cache for that prefix has been evicted. This can lead to suboptimal performance if the cache is no longer present on that instance.

  7. What is Prefix Aware Routing and how does it work?

    main

    Prefix aware routing is a feature in the vLLM Production Stack that ensures subsequent requests sharing the same prompt prefix are routed to the same vLLM instance.

    By routing requests with identical prefixes to the same instance, the system maximizes KV cache utilization. This reduces the need to recompute the KV cache for shared prefixes, thereby improving inference performance and reducing latency. The router detects these shared prefixes and schedules the requests accordingly, often visible in logs via LMCache indicating token retrieval from cache.

  8. What is vLLM Semantic Router?

    main

    The vLLM Semantic Router is an intelligent Mixture-of-Models (MoM) layer that acts as an Envoy External Processor (ExtProc). It semantically routes OpenAI API-compatible requests to the most suitable backend models using BERT-based or decoder-only LoRA classification, prompt guard, and semantic caching.

    Key capabilities include:

    • Auto-selection of models: Routes queries (math, creative writing, code, etc.) to the best-fit models.
    • Security & privacy: Provides PII detection, prompt guard, and safe routing.
    • Performance optimizations: Uses semantic caching and optimized tool selection to reduce latency and token usage.
    • Monitoring: Integrates with Console, Grafana, Prometheus, and tracing.

  9. Understand multi-round QA benchmark metrics

    main

    The benchmark evaluates the following performance metrics:

    • Queries Per Second (QPS): The average number of queries processed by the model per second.
    • Average Prompt Throughput: Tokens generated in the prompt per second.
    • Average Generation Throughput: Tokens generated as part of the response per second.
    • Average TTFT (Time to First Token): Average time taken for the model to generate the first token of a response.

  10. Configure Multi-GPU deployment with Tensor Parallelism

    main

    To deploy a model across multiple GPUs, you must enable tensor parallelism and configure shared memory (shmSize).

    Requirements for Multi-GPU:

    1. Set requestGPU to a value greater than 1.
    2. Set vllmConfig.tensorParallelSize to match the number of GPUs.
    3. Configure shmSize to provide sufficient shared memory for the processes used in tensor parallelism.

    Note: Currently, only tensor parallelism is supported for multi-GPU setups in this stack.

    servingEngineSpec:
  modelSpec:
  - name: "llama3"
    repository: "vllm/vllm-openai"
    tag: "latest"
    modelURL: "meta-llama/Llama-3.1-8B-Instruct"
    replicaCount: 1
    requestCPU: 10
    requestMemory: "16Gi"
    requestGPU: 2
    pvcStorage: "50Gi"
    pvcAccessMode:
      - ReadWriteOnce
    vllmConfig:
      enableChunkedPrefill: false
      enablePrefixCaching: false
      maxModelLen: 4096
      tensorParallelSize: 2
      dtype: "bfloat16"
      extraArgs: [ "--gpu-memory-utilization", "0.8"]
    hf_token: <YOUR HF TOKEN>
    shmSize: "20Gi"

  11. How semantic caching works in vLLM Router

    main

    The semantic cache module allows the vLLM router to avoid redundant LLM calls by returning cached responses for semantically similar requests.

    Mechanism:

    1. Embedding Generation: It uses sentence-transformers to generate embeddings for incoming chat messages.
    2. Vector Storage: Embeddings are stored in a FAISS vector database.
    3. Similarity Matching: For every new request, the router checks the FAISS index for existing entries that meet a configurable similarity threshold. If a match is found, the cached response is returned immediately instead of routing to the backend LLM.

  12. How Sleep Mode Aware Routing works

    main

    In the vLLM Production Stack, the router is aware of the sleep state of the underlying vLLM engines.

    • When an engine is sleeping: The router will not forward requests to it. If a user attempts to send a request to a sleeping engine via the /v1/completions endpoint, the router returns an error: "error": "Model <model_name> not found or vLLM engine is sleeping."
    • When an engine is awake: The router forwards requests normally to the engine.

    This mechanism prevents request timeouts and failures by ensuring the router only targets active engines, while allowing engines to release GPU memory when idle to optimize resource usage.