DOCSEARCHbyAlgolia
Submit Docs

GuideLLM Documentation

repository·main·Indexed 23 days ago

https://github.com/vllm-project/guidellm

An SLO-aware benchmarking and evaluation platform for optimizing real-world LLM inference. GuideLLM allows developers to simulate production workloads, generate realistic traffic patterns, and produce detailed performance reports (including TTFT and ITL) across text, image, audio, and video modalities. It supports OpenAI-compatible and vLLM-native servers, offering configurable load patterns such as synchronous, concurrent, and sweep profiles.

Tokens
55.7K
Snippets
128
Records
281
Agent score
75%

What's inside GuideLLM

  1. Overview of GuideLLM

    main

    GuideLLM is an SLO-aware (Service Level Objective) benchmarking and evaluation platform designed to optimize real-world LLM inference. It allows engineering and ML teams to simulate end-to-end interactions with OpenAI-compatible or vLLM-native servers using realistic workload patterns.

    Key capabilities include:

    • Latency & Token Statistics: Captures full distributions for Time To First Token (TTFT), Inter-Token Latency (ITL), and end-to-end behavior.
    • Traffic Pattern Generation: Supports synchronous, concurrent, and rate-based modes, including reproducible sweeps to find safe operating ranges.
    • Multimodal Support: Handles real and synthetic datasets for text, image, audio, and video.
    • Standardized Reporting: Produces exportable reports for dashboards and regression tracking.
    • High-Throughput Execution: Utilizes multiprocessing, threading, and async execution via a flexible CLI and API.

  2. Overview of GuideLLM capabilities

    main

    GuideLLM is an SLO-aware benchmarking and evaluation platform designed to optimize real-world LLM inference. It simulates end-to-end interactions with OpenAI-compatible and vLLM-native servers to help engineering teams understand system behavior, resource needs, and operational limits.

    Key Features

    • SLO-driven evaluation: Captures complete latency and token-level statistics (TTFT, ITL, and end-to-end distributions).
    • Configurable traffic patterns: Supports synchronous, concurrent, and rate-based modes, including sweep profiles for identifying safe operating ranges.
    • Multimodal support: Handles Text, Image, Audio, and Video using real or synthetic datasets (including HuggingFace datasets).
    • Extensible execution: High-throughput benchmarking via multiprocessing, threading, and async execution.
    • Standardized reporting: Exports results to console, json, csv, and html for analysis and regression tracking.

  3. Getting Started with GuideLLM

    main

    GuideLLM is a tool designed to evaluate and optimize Large Language Model (LLM) deployments. It helps developers find the optimal balance between performance (throughput and latency), resource utilization, and cost-effectiveness for real-world inference workloads.

    To use GuideLLM effectively, follow this workflow:

    1. Installation: Set up the GuideLLM environment.
    2. Server Setup: Launch an OpenAI-compatible inference server (e.g., using vLLM).
    3. Benchmarking: Run performance tests against the server under various load conditions.
    4. Analysis: Interpret the resulting data to optimize your deployment.

  4. Explore GuideLLM Key Guides

    main

    GuideLLM provides several specialized guides to help you configure, run, and analyze LLM benchmarks. Key areas of focus include:

    • Architecture: Understanding the modular design and component interactions.
    • Backends: Setting up OpenAI-compatible servers for benchmarking.
    • Datasets: Configuring synthetic, Hugging Face, or file-based data sources.
    • Metrics: Analyzing latency, throughput, and token-level performance.
    • Outputs: Customizing result reporting and output formats.
    • Service Level Objectives (SLOs): Defining and implementing SLOs/SLAs for reliability.
    • Over-Saturation Stopping: Automatically stopping benchmarks to prevent wasted compute.
    • Tool Calling: Benchmarking multi-turn tool calling workloads.
    • Multimodal Benchmarking: Setting up benchmarks for text+image, video, and audio tasks.
    • Troubleshooting: Resolving common errors.
    • v0.7.0 Migration: Instructions for migrating CLI commands from v0.6.0 to v0.7.0.

  5. Core components of GuideLLM

    main

    GuideLLM is composed of several specialized components:

    • Backend: An abstract interface for interacting with generative AI backends (e.g., OpenAI-compatible HTTP servers like vLLM). It handles request processing, response generation, and backend readiness validation.
    • RequestLoader: Sources data from iterables and generates formatted requests for the backend.
    • DatasetCreator: Loads data from local files, Hugging Face, or synthetic sources and converts them into HF dataset items for the RequestLoader.
    • Scheduler: Manages request scheduling, queues, and resource utilization using multiprocessing and asyncio.
    • RequestsWorker: A worker process that pulls requests from the scheduler, interacts with the backend, and returns results.
    • Benchmarker: Wraps multiple Scheduler invocations to manage multiple benchmarks and compiles them into a final report.
    • BenchmarkAggregator: Stores and compiles results from various benchmarks into a Benchmark object.

  6. Configure datasets using the kind pattern

    main

    GuideLLM uses a kind pattern to select dataset implementations and their parameters via the --data CLI flag. You can provide parameters as comma-separated key=value pairs for flat settings, or as a serialized JSON string for nested configurations. To combine multiple data sources, repeat the --data flag.

    Example command structure:

    guidellm run --data kind=<TYPE>,key=value,... [other options...]

  7. Understand GuideLLM console output

    main

    When a benchmark completes, GuideLLM displays three sections in the terminal to provide immediate feedback:

    1. Benchmarks Metadata: Summarizes the configuration used, including the target URL, model name, backend details, data source, token counts, and profile arguments (rate type, duration, etc.).
    2. Benchmarks Info: A summary table showing the benchmark type, start/end times, total duration, request counts (successful, incomplete, errored), and average token statistics.
    3. Benchmarks Stats: Detailed performance metrics divided into:
      • Throughput: Requests per second (RPS), request concurrency, and output/total tokens per second.
      • Latency: Mean, median, and p99 values for Request Latency, Time to First Token (TTFT), Inter-token Latency (ITL), and Time per output token.

    Note: The p99 values are critical for Service Level Objective (SLO) analysis as they represent worst-case performance.

  8. Understand GuideLLM Performance Metrics

    main

    GuideLLM provides several performance metrics to evaluate throughput, capacity, and responsiveness:

    Throughput and Capacity

    • Request Rate (Requests Per Second): The number of requests processed per second.
    • Request Concurrency: The number of requests being processed simultaneously.
    • Output Tokens Per Second: The average number of output tokens generated per second across all requests.
    • Total Tokens Per Second: The combined rate of prompt and output tokens processed per second.

    Latency and Responsiveness

    • Request Latency: Total time taken to process a single request from start to finish.
    • Time to First Token (TTFT): Time taken to generate the very first token of the output. Crucial for user-facing responsiveness.
    • Inter-Token Latency (ITL): The average time between consecutive output tokens (excluding the first token). Measures generation smoothness.
    • Time Per Output Token: The average time taken to generate each output token, including the first one.

  9. How GuideLLM components work together

    main

    GuideLLM uses a modular architecture to simulate real-world inference workloads for LLM evaluation. The workflow follows a pipeline from data creation to benchmark reporting:

    1. DatasetCreator loads data (local, Hugging Face, or synthetic) and converts it into HF dataset items.
    2. RequestLoader streams these items and formats them into backend-compatible requests.
    3. Scheduler manages the distribution of these requests using multiprocessing and asyncio to maximize throughput.
    4. RequestsWorker processes requests by interacting with the Backend.
    5. Backend (e.g., an OpenAI-compatible server like vLLM) executes the actual inference.
    6. BenchmarkAggregator collects results from the workers.
    7. Benchmarker orchestrates multiple benchmark runs and compiles the final report.

  10. Understand output token limits on tool-call turns

    main

    When performing tool-calling benchmarks, GuideLLM automatically modifies certain generation settings on tool-call turns to ensure compatibility with constrained decoding (e.g., vLLM's grammar-based decoding).

    On tool-call turns, the following settings are automatically removed:

    • ignore_eos: Removed because it conflicts with the grammar's terminal state (EOS is required to signal the end of valid JSON).
    • stop=None: Removed because the tool-call parser relies on specific stop sequences (like <|eot_id|>).
    • max_tokens / max_completion_tokens: Removed to prevent truncation mid-JSON, which would corrupt the conversation history.

    Consequence: The length of tool-call turn outputs is determined by the model and the tool schema rather than the configured output_tokens. These turns typically result in shorter outputs (roughly 20–80 tokens) compared to plain text turns. Plain text turns are unaffected and continue to respect output_tokens and ignore_eos normally.

  11. Understand GuideLLM Token Metrics

    main

    Token metrics allow you to analyze the complexity of your workloads and the efficiency of the model's generation:

    • Prompt Tokens and Counts: The number of tokens in the input prompts. Use this to understand input complexity.
    • Output Tokens and Counts: The number of tokens generated by the LLM. Use this to evaluate output length and its impact on latency and resource usage.

  12. Override Constraints for Sub-benchmarks

    main

    When running profiles with multiple strategies (like sweep, concurrent with multiple streams, or constant with multiple rate values), you can use the --override option to apply different constraint values to each individual strategy.

    Example: To run 5 strategies in a sweep where each has a different duration, use --override constraint[0].seconds 10,20,10,15,20. If you provide fewer values than strategies, the final value is reused for the remaining strategies.

    --override constraint[0].seconds 10,20,10,15,20