DOCSEARCHbyAlgolia
Submit Docs

vLLM Documentation and Benchmarking Tools

repository·main·Indexed Apr 15, 2026

https://github.com/vllm-project/vllm

vLLM is a fast and easy-to-use library for LLM inference and serving, delivering state-of-the-art throughput using PagedAttention, continuous batching, and chunked prefill. This documentation set covers the vLLM CLI, offline inference, OpenAI-compatible API, and custom logits processors. It includes comprehensive benchmarking tools for automated server parameter tuning (max-num-seqs, max-num-batched-tokens), throughput maximization, latency constraints, and prefix cache hit rate optimization. Features include batch execution via batch_auto_tune.sh, custom workload definition using batch specification grammar, parameter sweeps for optimization (e.g., num_kv_splits, reorder_batch_threshold), and DeepGEMM kernel benchmarks for Hopper GPUs. The suite also provides Grafana dashboards for monit…

Tokens
33.7K
Snippets
79
Records
144
Agent score
100%

What's inside vllm

  1. Overview: Grafana Dashboards for vLLM Monitoring

    main

    This directory provides pre-configured Grafana dashboards to monitor vLLM performance and query statistics. The dashboards help visualize latency, throughput, request volume, and key performance indicators for your vLLM service.

    Available Dashboards:

    • performance_statistics.json: Tracks performance metrics including latency and throughput.
    • query_statistics.json: Tracks query performance, request volume, and KPIs.

    Requirements:

    • Grafana 8.0+
    • Prometheus data source configured in Grafana
    • vLLM deployment with Prometheus metrics enabled

    Sources: examples/online_serving/dashboards/grafana/README.md

  2. Benchmark with Custom Datasets

    main

    Benchmark against custom datasets not supported by default. Your data must be in .jsonl format with a prompt field per entry.

    Custom Text Dataset:

    1. Create data.jsonl:
       {"prompt": "What is the capital of India?"}
   {"prompt": "What is the capital of Iran?"}
   {"prompt": "What is the capital of China?"}
    1. Start the server:
       vllm serve meta-llama/Llama-3.1-8B-Instruct
    1. Run benchmark:
       vllm bench serve --port 9001 --save-result --save-detailed \
     --backend vllm \
     --model meta-llama/Llama-3.1-8B-Instruct \
     --endpoint /v1/completions \
     --dataset-name custom \
     --dataset-path <path-to-your-data-jsonl> \
     --custom-skip-chat-template \
     --num-prompts 80 \
     --max-concurrency 1 \
     --temperature=0.3 \
     --top-p=0.75 \
     --result-dir "./log/"

    Custom Multimodal Dataset: For image/video data, use CustomMMDataset with prompt and image_files fields:

    {"prompt": "How many animals are present in the given image?", "image_files": ["/path/to/image/folder/horsepony.jpg"]}

    Run with:

    vllm bench serve --save-result --save-detailed \
  --backend openai-chat \
  --model Qwen/Qwen2-VL-7B-Instruct \
  --endpoint /v1/chat/completions \
  --dataset-name custom_mm \
  --dataset-path <path-to-your-mm-data-jsonl> \
  --allowed-local-media-path /path/to/image/folder

    Note: Use --custom-skip-chat-template if your data already includes chat templates.

    vllm bench serve --port 9001 --save-result --save-detailed \
  --backend vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --endpoint /v1/completions \
  --dataset-name custom \
  --dataset-path <path-to-your-data-jsonl> \
  --custom-skip-chat-template \
  --num-prompts 80

    Sources: docs/benchmarking/cli.md

  4. Install vLLM for CPU (x86, ARM, Apple Silicon, IBM Z)

    main

    To install vLLM on CPU hardware, follow the specific instructions in the CPU installation guide. Supported CPU platforms include:

    • Intel/AMD x86: Standard x86_64 architecture.
    • ARM AArch64: ARM-based architecture.
    • Apple silicon: M1/M2/M3 chips on macOS.
    • IBM Z (S390X): IBM mainframe architecture.

    Refer to docs/getting_started/installation/cpu.md for detailed steps, including environment setup and package installation commands specific to your CPU architecture.

    Sources: docs/getting_started/installation/README.md

  5. vLLM Usage Patterns and Entry Points

    main

    This repository provides the vLLM library for high-performance LLM serving. Before using any feature, ensure vLLM is installed for your target device (Python or Docker environment) following the Installation Guide.

    Once installed, vLLM supports three primary usage patterns:

    1. Inference and Serving: Run a single instance of a model for offline inference. See the Offline Inference Guide.
    2. Deployment: Scale up model instances for production environments using Docker. See the Docker Deployment Guide.
    3. Training: Train or fine-tune models using Reinforcement Learning from Human Feedback (RLHF). See the RLHF Training Guide.

    Sources: docs/usage/README.md

  6. Install Third-Party Hardware Plugins

    main

    vLLM supports third-party hardware plugins that live outside the main vllm repository. These plugins follow the Hardware-Pluggable RFC.

    To find supported hardware plugins, visit the Universal Compatibility - Hardware page on the vLLM website.

    If you need to add support for new hardware, contact the vLLM team via Slack or Email.

    Sources: docs/getting_started/installation/README.md

  7. Benchmark Best Practices and Tips

    main

    Follow these tips to ensure accurate and stable benchmark results:

    1. Warmup matters: Use --warmup-iters 10 to stabilize performance metrics before measuring.
    2. Multiple repeats: Use --repeats 20 to reduce variance in results.
    3. Save results: Always use --output-csv or --output-json to persist benchmark data.
    4. Test incrementally: Start with --num-layers 1 --repeats 1 to verify setup before running full benchmarks.
    5. Extended grammar: Leverage the batch specification grammar for complex patterns like spec decode and chunked prefill.
    6. Parameter sweeps: Use --sweep-param and --sweep-values to systematically find optimal configuration values.

    Sources: benchmarks/attention_benchmarks/README.md

  8. Prerequisites for OpenAI Batch Inference

    main

    Before running batch inference, ensure the following prerequisites are met:

    1. Model Access: The examples use meta-llama/Meta-Llama-3-8B-Instruct. You must:

      • Create a Hugging Face user access token.
      • Install the token on your machine by running hf auth login.
      • Visit the model card and agree to the terms to gain access to the gated model.

    2. Dependencies:

      • For embeddings: Ensure vllm >= 0.5.5.
      • For S3 integration: Install awscli (pip install awscli) and boto3 (pip install boto3).

    3. Input File: Create a JSONL file with your batch requests. You can download the example file:

       wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai_batch/openai_example_batch.jsonl

    Sources: examples/offline_inference/openai_batch/README.md

  9. Benchmark KV Cache Offloading with Multi-Turn Conversations

    main
    This guide explains how to benchmark vLLM's KV cache offloading capabilities using synthetic or real multi-turn conversations. The benchmark measures metrics like Time to First Token (TTFT), Time Per Output Token (TPOT), and overall latency under concurrent load.

  10. Install Dashboards via Manual Import

    main

    The recommended way to use these dashboards is to manually import the JSON configurations into your Grafana instance.

    Steps:

    1. Navigate to your Grafana instance.
    2. Click the + icon in the sidebar.
    3. Select Import.
    4. Copy the JSON content from the dashboard files (performance_statistics.json or query_statistics.json) or upload the JSON files directly.

    Once imported, the dashboards will be available in your Grafana UI for monitoring vLLM metrics.

    Sources: examples/online_serving/dashboards/grafana/README.md

  11. Run Helm Chart Unit Tests

    main

    The chart includes unit tests using helm-unittest to verify template rendering and logic. Install the plugin and run the tests to ensure the chart behaves as expected before deployment.

    # Install the helm-unittest plugin
helm plugin install https://github.com/helm-unittest/helm-unittest

# Run the unit tests
helm unittest .

    Sources: examples/online_serving/chart-helm/README.md

  12. Setup

    main
    1. Install vLLM in your usual fashion.
    2. Clone and install DeepGEMM from source in a separate directory:
    git clone --recursive https://github.com/deepseek-ai/DeepGEMM
cd DeepGEMM
python setup.py install
uv pip install -e .