DOCSEARCHbyAlgolia
Submit Docs

GenAI Bench

repository·main·Indexed 43 days ago

https://github.com/sgl-project/genai-bench

A benchmark tool for comprehensive token-level performance evaluation of Large Language Model (LLM) serving systems. Version 0.0.5 supports text and image generation, providing CLI tools for benchmarking, real-time UI monitoring, automated Excel reporting, and performance visualization of metrics like throughput and latency.

Tokens
35.4K
Snippets
93
Records
129
Agent score
56%

What's inside GenAI Bench

  2. Overview of GenAI Bench features

    main

    GenAI Bench provides a comprehensive suite of tools for LLM benchmarking:

    • CLI Tool: Validates inputs and manages benchmark execution.
    • Live UI Dashboard: Provides real-time monitoring of progress, logs, and metrics.
    • Rich Logs: Automatically saves logs to both the terminal and a file upon completion.
    • Experiment Analyzer: Generates Excel reports containing pricing and raw metrics (throughput, latency like TTFT, E2E, TPOT, error rates, and RPS). It also supports generating visualizations with flexible plot configurations (defaulting to a 2x4 grid).

  3. Coding guidelines for genai-bench

    main

    When contributing to genai-bench, adhere to the following standards:

    • Python Version: 3.10-3.13.
    • Style Guide: Follow the Google Python style guide and PEP 8.
    • Type Safety: Use type hints throughout the codebase.
    • Documentation: Write docstrings for all public APIs.
    • Design: Keep functions focused and small.
    • Testing: Add tests for every new feature.
    • Formatting/Linting: Refer to the Makefile for available linting and formatting commands.

  5. How tasks and modalities work in genai-bench

    main

    Tasks are driven by two core concepts:

    1. Input Modality: The type of input data the task operates on (e.g., text or images).
    2. Output Modality: The type of output the task generates (e.g., text, embeddings, or audio).

    When you select a task via the --task flag, genai-bench automatically selects the appropriate internal components, including the correct sampler (TextSampler or ImageSampler) and the corresponding request type (e.g., UserChatRequest, UserEmbeddingRequest).

  6. Understand Single Request Level Metrics

    main

    Single-request level metrics capture token-level performance for an individual request, providing insights into server efficiency. Each metric includes standard statistics: percentile, min, max, stddev, and mean.

    Key Latency and Speed Metrics

    • TTFT (Time to First Token): The initial response time when the first output token is generated (input stage latency). Measured in seconds.
    • End-to-End Latency: Total time from query submission to receiving the full response, including network latencies. Measured in seconds.
    • TPOT (Time Per Output Token): The average time between two subsequent generated tokens. Measured in seconds.
    • Output Latency: Time taken to receive the full response after the first token is generated. Measured in seconds.
    • Output Inference Speed: The rate of token generation per second for a single request. Measured in tokens/second.
    • Input Throughput: The overall throughput of the input process. Measured in tokens/second.
    • Output Throughput: The throughput of output generation for a single request. Measured in tokens/second.

    Token Counting Metrics

    • Num of Input Tokens: Number of prompt tokens.
    • Num of Output Tokens: Number of completion tokens.
    • Num of Request Tokens: Total tokens processed (input + output).
    • Num of Reasoning Tokens: Number of reasoning tokens outputted (always $\le$ output tokens).

    Task-Specific Metrics

    • Embedding Tasks: Since there is no output stage, output metrics are set to 0.
    • Text-to-Speech (TTS) Tasks:
      • TTFB (Time to First Byte): Replaces TTFT; the time until the first audio chunk arrives.
      • Audio Throughput: Replaces token-based output metrics; measures the rate of audio data generated per second. Measured in bytes/second.

  7. Understand Aggregated Metrics

    main

    Aggregated level metrics summarize single-request metrics from an entire run, which is defined by a specific traffic scenario and number of concurrency. These metrics provide insights into server capacity and performance under pressure.

    Throughput Metrics

    • Mean Input Throughput: Average input tokens processed by the model in one run with multiple concurrent requests. Measured in tokens/second.
    • Mean Output Throughput: Average output tokens processed by the model in one run with multiple concurrent requests. Measured in tokens/second.
    • Total Tokens Throughput: Average throughput of all tokens (input + output) processed during the run. Measured in tokens/second.
    • Requests Per Minute: The number of requests processed by the model per minute.

  8. Define traffic scenarios for request shaping

    main

    Traffic scenarios shape the requests during a benchmark run by defining token distributions, budgets, or modality properties. You can pass one or more scenarios using the --traffic-scenario flag. If you provide a --dataset-path without any scenarios, the tool defaults to dataset mode, which uses raw entries from your dataset without token shaping.

    Supported Scenario Formats

    TypeFormatDescription
    TextD(in,out)Deterministic: Exact in input and out output tokens.
    N(m_in,s_in)/(m_out,s_out)Normal: Input/output tokens follow a normal distribution with mean and stddev.
    U(min_in,max_in)/(min_out,max_out)Uniform: Input/output tokens follow a uniform distribution.
    U(max_in,max_out)Uniform: Only input tokens are shaped.
    EmbeddingsE(tokens_per_doc)Fixed tokens per document.
    Re-RankR(tokens_per_doc,tokens_per_query)Fixed tokens for documents and queries.
    AudioA(num_input_chars)Fixed number of input characters.
    ImageI(width,height)Fixed image dimensions.
    I(width,height,num_images)Fixed dimensions and number of images.
    SpecialdatasetUse raw dataset entries without shaping.

    Note: Scenario strings are validated; mistyping a format will trigger an error showing supported types.

    # Example: Running with multiple text scenarios (Normal and Deterministic)
genai-bench \
  --task text-to-text \
  --dataset-path prompts.txt \
  --traffic-scenario "N(480,240)/(300,150)" \
  --traffic-scenario "D(100,1000)" \
  --api-backend openai \
  --api-base http://localhost:8000 \
  --api-model-name gpt-test \
  --model-tokenizer gpt2

  9. Supported benchmark tasks in GenAI Bench

    main

    GenAI Bench supports several task types depending on your evaluation needs:

    • text-to-text: Benchmarks generating text output from text input (e.g., Chat, QA).
    • text-to-embeddings: Benchmarks generating embeddings from text input (e.g., Semantic search).
    • image-text-to-text: Benchmarks generating text from images and text prompts (e.g., Visual question answering).
    • image-to-embeddings: Benchmarks generating embeddings from images (e.g., Image similarity).

  10. Use dataset mode for raw sampling

    main

    Dataset mode is used when you want to sample raw entries from a dataset without applying token shaping or length constraints. This mode is automatically enabled if you provide a --dataset-path but omit the --traffic-scenario flag. You can also explicitly set it using --traffic-scenario dataset.

    Behavior by Task

    • text-to-text: Picks a single line from the dataset; max_tokens=None and ignore_eos=False are used to avoid shaping.
    • text-to-embeddings: Samples batch_size lines per request.
    • text-to-rerank: Samples one query and batch_size documents.
    • text-to-image: Picks a single line as the prompt; size/quality use server defaults.
    • text-to-speech: Picks a single line as input text; no character-length shaping is applied.
    • image-text-to-text / image-to-embeddings: Samples images (defaults to 1 image per request).
    # Run using dataset-only mode (no token shaping)
genai-bench \
  --task text-to-text \
  --dataset-path data.csv \
  --api-backend openai \
  --api-base http://localhost:8000 \
  --api-model-name gpt-test \
  --model-tokenizer gpt2

  11. Understand GenAI-bench authentication separation

    main

    GenAI-bench separates authentication into two distinct categories to allow cross-cloud workflows (e.g., benchmarking a model on one cloud while storing results on another):

    1. Model Authentication: Used for accessing LLM endpoints (e.g., OpenAI, AWS Bedrock, GCP Vertex AI).
    2. Storage Authentication: Used for uploading benchmark results to storage services (e.g., AWS S3, OCI Object Storage).

    This separation enables flexible benchmarking pipelines where the compute provider and the data storage provider do not need to be the same.

  12. Simulate bursty vs uniform traffic patterns

    main

    GenAI Bench uses the --traffic-scenario flag to control the variability of token lengths, which can be used to simulate different traffic patterns:

    • Bursty Traffic: Use distributions with high variance, such as a Normal distribution with a large standard deviation.
      • Example: --traffic-scenario "N(512,256)/(512,256)"
    • Uniform/Consistent Traffic: Use deterministic distributions or distributions with very low variance.
      • Example: --traffic-scenario "D(512,512)"
    # Bursty pattern example
genai-bench benchmark \
    --api-backend vllm \
    --api-base http://127.0.0.1:8000 \
    --api-model-name vllm-model \
    --model-tokenizer NousResearch/Hermes-3-Llama-3.1-8B \
    --task text-to-text \
    --max-requests-per-run 500 \
    --max-time-per-run 30 \
    --traffic-scenario "N(512,256)/(512,256)" \
    --num-concurrency 8 \
    --server-engine "vLLM"