DOCSEARCHbyAlgolia
Submit Docs

vLLM: High-Throughput LLM Inference and Serving Engine

repository·main·Indexed Aug 11, 2026

https://github.com/vllm-project/vllm

A high-performance library for LLM inference and serving optimized for high throughput using PagedAttention and continuous batching. It includes a comprehensive benchmarking suite for attention backends (including MLA), automated server parameter tuning via auto_tune.sh, and specialized kernels like Machete for NVIDIA Hopper architectures.

Tokens
328.3K
Snippets
871
Records
1.3K
Agent score
73%

What's inside vLLM

  1. Overview of Model Runner V2 (MRV2) design

    main

    Model Runner V2 (MRV2) is a redesigned execution core for vLLM, built from first principles to improve efficiency, modularity, and asynchronous performance. It replaces the V1 architecture by introducing several key improvements:

    • Persistent Batching: Decouples persistent state from per-step inputs to simplify bookkeeping and reduce CPU overhead.
    • Async-First Architecture: Designed to overlap CPU preparation and GPU execution without blocking or explicit synchronization barriers.
    • GPU-Native Operations: Uses Triton kernels for input metadata preparation (e.g., input_ids, positions) and sampling, reducing Python bottlenecks.
    • Staged Writes: Uses StagedWriteTensor for efficient, non-blocking updates of large tensors like block tables.
    • Explicit CUDA Graph Management: Uses a dedicated CUDAGraphManager for more predictable lifecycle and execution control.
    • Improved Speculative Decoding: Supports fused multi-step draft decoding and more efficient sampling via Triton kernels.

  2. Overview of LLM Compressor

    main

    LLM Compressor is a library designed to optimize Large Language Models (LLMs) for deployment with vLLM. It reduces the memory footprint, lowers inference costs, and increases throughput by quantizing model weights and activations to lower-precision data types (e.g., FP8, INT8, INT4).

    Key capabilities include:

    • Quantization Algorithms: Supports AWQ, GPTQ, AutoRound, Round-to-Nearest, QuIP, and SpinQuant-style transforms.
    • Quantization Methods: Supports FP8, INT8, INT4, NVFP4, MXFP4, and mixed-precision quantization.
    • Specialized Quantization: Includes support for KV cache and attention quantization.
    • Deployment: Produces models in the compressed-tensors format for seamless integration with vLLM and maintains compatibility with Hugging Face Hub models.

  3. Overview of vLLM Benchmark Suites

    main

    vLLM offers several tools for performance testing and evaluation of LLM serving:

    • Benchmark CLI: Use vllm bench CLI tools and specialized scripts for interactive performance testing.
    • Parameter Sweeps: Automate vllm bench runs across multiple configurations to assist with optimization and tuning.
    • Performance Dashboard: An automated CI system that publishes benchmark results on every commit.
    • vLLM Recipes: Convert vLLM Recipes into config.yaml and env.sh files using the Recipe conversion tool to facilitate benchmarking.

  4. Overview of vLLM features and capabilities

    main

    vLLM is a high-throughput library for LLM inference and serving. Key technical features include:

    Performance Optimizations

    • PagedAttention: Efficient management of attention KV memory.
    • Continuous Batching: Handles incoming requests with chunked prefill and prefix caching.
    • Execution Speed: Uses piecewise and full CUDA/HIP graphs, and optimized kernels (FlashAttention, FlashInfer, Triton, etc.).
    • Speculative Decoding: Supports n-gram, suffix, EAGLE, and DFlash.
    • Quantization Support: Includes FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, and more.

    Flexibility and Integration

    • Model Support: Seamless integration with Hugging Face; supports 200+ architectures including Decoder-only, MoE, Hybrid (Mamba), Multi-modal (LLaVA), and Embedding models.
    • Serving: Provides an OpenAI-compatible API server, Anthropic Messages API, and gRPC support.
    • Distributed Inference: Supports Tensor, pipeline, data, expert, and context parallelism.
    • Advanced Features: Structured outputs (via xgrammar or guidance), tool calling, reasoning parsers, and multi-LoRA support.

  5. Overview of Speculators for LLM acceleration

    main

    Speculators is a library designed to accelerate LLM inference via speculative decoding. It integrates with vLLM to reduce latency and improve throughput by using a smaller 'draft' model to predict multiple tokens ahead, which are then verified in parallel by the primary model.

    Key capabilities include:

    • Offline training data generation: Uses vLLM to generate hidden states and save data samples to disk for training draft models.
    • Draft model training: End-to-end training support for single and multi-layer draft models, including both non-MoE and MoE architectures.
    • Standardized format: Uses a Hugging Face-compatible format for defining speculative models, with conversion tools available for external research repositories.
    • vLLM Integration: Designed for direct deployment into vLLM for production-grade inference.

  6. Overview of vLLM benchmark types

    main

    vLLM provides several categories of benchmark scripts and utilities to evaluate performance and specific features:

    • Serving benchmarks: Used for testing online inference performance, specifically focusing on latency and throughput.
    • Throughput benchmarks: Used for testing offline batch inference performance.
    • Specialized benchmarks: Tools designed to test specific vLLM features, including:
      • Structured output
      • Prefix caching
      • Long document QA
      • Request prioritization
      • Multi-modal inference
    • Dataset utilities: A framework for loading and sampling from various datasets such as ShareGPT, HuggingFace datasets, and synthetic data.

  7. What is vLLM?

    main

    vLLM is a high-throughput library designed for LLM inference and serving. It is optimized for speed and efficiency through several core technologies:

    Performance Features

    • PagedAttention: Efficient management of attention KV memory.
    • Continuous Batching: Handles incoming requests with chunked prefill and prefix caching.
    • Optimized Kernels: Supports FlashAttention, FlashInfer, Triton, and specialized GEMM/MoE kernels.
    • Speculative Decoding: Includes n-gram, suffix, EAGLE, and DFlash.
    • Quantization Support: Supports FP8, INT8, INT4, GPTQ/AWQ, GGUF, and more.

    Flexibility and Integration

    • Model Support: Seamlessly integrates with Hugging Face models, supporting 200+ architectures including Decoder-only (Llama, Qwen), MoE (Mixtral, DeepSeek-V3), Multi-modal (LLaVA), and Embedding models.
    • Distributed Inference: Supports Tensor, pipeline, data, expert, and context parallelism.
    • APIs: Provides an OpenAI-compatible API server, Anthropic Messages API, and gRPC support.
    • Hardware Support: Works with NVIDIA, AMD, and Intel GPUs, as well as CPUs (x86/ARM/PowerPC) and various hardware plugins (TPU, Gaudi, etc.).

  9. Get started with vLLM usage patterns

    main

    vLLM can be used for several distinct workloads depending on your requirements. Before using any pattern, ensure vLLM is installed for your specific device in either a Python or Docker environment.

    Supported usage patterns include:

    1. Inference and Serving: Running a single instance of a model for inference tasks.
    2. Deployment: Scaling up model instances for production environments (typically via Docker).
    3. Training: Performing training or fine-tuning tasks, such as RLHF (Reinforcement Learning from Human Feedback).

  10. Explore vLLM usage examples

    main

    vLLM provides a variety of categorized examples to help you implement different inference patterns. Depending on your use case, you can find specialized code for:

    • Core Inference: Minimal examples for offline inference and online serving (basic/), and text generation including multimodal models (generate/).
    • Specialized Tasks: Embedding, classification, scoring, and reward models (pooling/), speech transcription, translation, and real-time audio (speech_to_text/), and reasoning (reasoning/).
    • Advanced Features: Demonstrations of speculative decoding, LoRA, structured outputs, automatic prefix caching, prompt embedding, pause/resume, batch invariance, KV events, and data parallelism (features/).
    • Agentic & Application Patterns: Function/tool calling (tool_calling/), chatbots, RAG (Retrieval-Augmented Generation), and custom API servers (applications/).
    • Scaling & Production: Production deployment (deployment/), scalable serving with Ray (ray_serving/), Disaggregated P/D (Prefill/Decoding) inference with various KV cache connectors like LMCache or Mooncake (disaggregated/), and Token In <> Token Out API Server (scale_out/).
    • Optimization & Training: Reinforcement learning (rl/) and observability via metrics, logging, tracing (OpenTelemetry), and dashboards (observability/).