DOCSEARCHbyAlgolia
Submit Docs

AIPerf Documentation

repository·main·Indexed 19 days ago

https://github.com/ai-dynamo/aiperf

A comprehensive benchmarking tool for measuring the performance of generative AI models. AIPerf provides detailed metrics, real-time TUI dashboards, and performance reports for various inference solutions, including OpenAI and NIM APIs. It supports multiple benchmarking modes such as concurrency, request-rate, and trace replay, and features a plugin system for custom endpoints, datasets, and metrics. Version 0.12.0 includes a TCO calculator for NVIDIA NIM LLM deployments and integrations with MLflow, OpenTelemetry, and Weights & Biases.

Tokens
324.6K
Snippets
636
Records
1.1K
Agent score
65%

What's inside aiperf

  1. Overview of AIPerf Profile Export Files

    main

    AIPerf generates several output files after a benchmark run, each serving a different analysis purpose:

    • inputs.json: Contains the complete input dataset with formatted payloads for every request.
    • profile_export.jsonl: A JSON Lines file containing per-request metric records (one record per line).
    • profile_export_aiperf.json: A single JSON object containing aggregated statistics (min, max, mean, percentiles) and the user configuration.
    • profile_export_aiperf.csv: The same aggregated statistics as the JSON version, formatted for spreadsheets.

  2. Overview of Server Metrics Collection in AIPerf

    main

    AIPerf automatically collects metrics from Prometheus-compatible endpoints exposed by LLM inference servers and serving frontends (such as vLLM, SGLang, TRT-LLM, Dynamo, and Triton).

    Key Features

    • Auto-discovery: Automatically finds the /metrics endpoint on the provided server URL.
    • Collection Frequency: Scrapes metrics every 333ms during the benchmark.
    • Output Formats: Generates aggregated statistics in JSON and CSV, cumulative deltas in Parquet, and optional raw time-series snapshots in JSONL.
    • Customization: Supports adding custom endpoints via --server-metrics and controlling output formats via --server-metrics-formats.

  3. AIPerf Core Features and Architecture

    main

    AIPerf is a high-performance benchmarking tool designed to measure generative AI model performance.

    Key Capabilities:

    • Scalable Architecture: Uses a multiprocess architecture with 10 services communicating via ZMQ.
    • UI Modes:
      • dashboard: Real-time Terminal User Interface (TUI).
      • simple: Progress bar-based display.
      • none: Headless mode for automated environments.
    • Benchmarking Modes: Supports concurrency, request-rate, request-rate with max concurrency, and trace replay.
    • Extensibility: Features a plugin system for custom endpoints, datasets, transports, and metrics.
    • Data Support: Supports public datasets like ShareGPT and custom formats.
    • Supported APIs: OpenAI (chat completions, completions, embeddings, audio, images) and NIM (embeddings, rankings).

  4. Core Components of AIPerf Architecture

    main

    AIPerf is organized into several specialized modules that coordinate to execute and analyze benchmarks:

    • System Controller: The central orchestrator managing component initialization, task execution, resource allocation, and error handling.
    • Dataset Manager: Handles input data (JSONL, CSV, synthetic, or trace replay). It uses memory-mapped files to allow workers to access data without message-passing overhead.
    • Timing Manager: Controls request scheduling using a credit-based system to support fixed schedules, request rates, or user-centric rates.
    • Worker Manager: Manages the pool of worker processes.
    • Workers: Processes that send HTTP requests to the inference server, track conversation state, and measure response times.
    • Record Processor: Parses raw inference responses to extract metrics (latency, tokens, correctness) and computes derived metrics like TTFT and ITL.
    • Records Manager: Aggregates and stores results, providing interfaces for querying and exporting data to CSV, JSON, or Parquet.
    • GPU Telemetry Manager: Collects GPU metrics (power, utilization, memory) via pluggable collectors (DCGM, PyNVML, or amdsmi).
    • Server Metrics Manager: Collects Prometheus-compatible metrics from the inference server endpoints.

  5. Deployment modes and scalability

    main

    AIPerf is designed for high-throughput benchmarking with the following deployment characteristics:

    Supported Deployment Model:

    • Multiprocess Mode: The default mode where each service runs as a separate process on a single node. This is intended for single-node deployments.

    Scalability & Design:

    • Stateless Workers: Workers maintain no shared state; they only keep local conversation context for multi-turn requests.
    • Stateless Services: All service state is ephemeral and can be reconstructed from configuration.
    • Horizontal Scaling: Supports scaling workers and record processors horizontally.

    Note on Kubernetes: While code paths for Kubernetes exist, distributed Kubernetes execution is not currently supported in this version. Do not attempt to use Kubernetes as a distributed service manager.

  6. Architecture Overview of AIPerf

    main
    AIPerf is a modular benchmarking framework designed to measure AI inference performance by generating load, collecting metrics, and analyzing throughput, latency, and resource utilization. The system uses a three-plane architecture to separate orchestration, execution, and analysis, allowing for horizontal scaling by adding more workers.

  7. What is a Weka trace and how is it structured?

    main

    A Weka trace is a JSON object representing a coding conversation. It consists of an ordered list of requests with the following types:

    • type: "n": Normal API calls.
    • type: "s": Streaming API calls.
    • type: "subagent": Markers for nested sub-conversations.

    Key Features:

    • KV-Cache Awareness: Requests carry hash_ids (cache-block identifiers) to simulate cache reuse.
    • Subagent Topology: Subagents are replayed as concurrent child sessions. AIPerf uses SPAWN and SPAWN_JOIN markers to manage the parent-child relationship. If a subagent has no following parent turn, it is treated as is_background=True (the parent does not wait).

  8. Overview of Custom Dataset Types in AIPerf

    main

    AIPerf supports several custom dataset types for benchmarking LLMs with your own data. Choosing the right type depends on whether you need multi-turn conversations, timing control, or random sampling:

    Dataset TypeBest ForMulti-TurnTiming ControlRandom Sampling
    Single TurnIndependent single requestsNoYesNo
    Multi TurnConversations with contextYesYes (per turn)No
    Random PoolLoad testing with varietyNoNoYes
    Mooncake / Bailian / Baseten TraceProduction trace replayYesYesNo

    Common Features:

    • Single Turn, Multi Turn, and Random Pool support client-side batching and automatic media handling (local files are converted to base64, remote URLs are sent directly).
    • Trace Replay requests are text-only; client-side batching and media handling do not apply.

  9. What is Adaptive Scale and when to use it

    main

    Adaptive scale is a single-run load controller designed to find and sustain an SLA (Service Level Agreement) boundary. Instead of running multiple independent sweeps or searches, it starts at a low load, increases it as long as SLA filters pass, and then sustains the load near the last successful boundary.

    When to use Adaptive Scale:

    • Use it when you want a single benchmark invocation to push a service until a latency or throughput constraint fails, then keep pressure near that edge.

    Comparison with other modes:

    • adaptive_search: Use for offline Bayesian optimization across multiple runs.
    • sweeps: Use for a fixed experiment grid.
    • fixed ramps: Use when you have a known, repeatable schedule.
    • static concurrency/request rate: Use for a single fixed load point.

  10. What is gradual ramping in AIPerf?

    main

    Gradual ramping is a technique used in AIPerf to increase concurrency and request rates smoothly over time instead of applying the full target load immediately.

    Why use it? Immediate high loads can cause:

    • Connection storms: Hundreds of simultaneous connections overwhelming the server.
    • Memory spikes: Sudden KV-cache allocations causing Out-of-Memory (OOM) errors or performance degradation.
    • Misleading metrics: Cold-start effects (caches, JIT, connection pools) polluting steady-state measurements.

    Benefits:

    • Allows the server to warm up gradually.
    • Enables early detection of capacity limits.
    • Provides cleaner measurements once the steady state is reached.

  11. What is the AgentX MVP scenario and its rules?

    main

    The inferencex-agentx-mvp scenario is a specialized benchmark recipe that implements the SemiAnalysis AgentX-MVP specification. It is designed to move beyond synthetic 1-turn prompts by using real-world coding traces (from the Weka corpus) that include parent-subagent interactions (SPAWN/JOIN patterns).

    When you use the --scenario inferencex-agentx-mvp flag, AIPerf enforces the following behaviors to ensure comparability between different hardware/software setups:

    • Preserve Request Timing: Maintains the original timing of requests from the trace.
    • Idle Limit: Ensures the system is never left idle for more than 10 seconds.
    • Full Generation: The server must be allowed to generate complete responses without early stopping.
    • Cache Warm-up: Performs a warm-up phase before measurement begins.
    • Rule Enforcement: AIPerf will reject any CLI flags that conflict with these locked settings.
    • Validation: A submission_valid field is added to the JSON output to confirm the run followed these rules.

  12. Configure Multi-Phase Workflows

    main

    Multi-phase workflows allow you to define an ordered sequence of load windows within a single benchmark run. This is useful for scenarios like cache warmup, baseline profiling, stress testing (e.g., cancellation storms), and recovery monitoring.

    Each phase in the phases list requires a name and a kind.

    • name: A unique identifier used for logs, artifact directories, and sweep paths. Must follow the regex ^[A-Za-z_][A-Za-z0-9_-]*$.
    • kind: Defines the semantic role of the phase:
      • warmup: These phases are excluded from aggregate profiling results. Use these for cache priming or system settling.
      • profiling: These phases contribute to the benchmark results. A workflow must contain at least one profiling phase.

    Note on Canonical Names: If you name a phase warmup, its kind defaults to warmup. If you name it profiling, its kind defaults to profiling. For any other name, you must explicitly set the kind field.

    benchmark:
  phases:
    - name: warmup
      type: concurrency
      concurrency: 8
      duration: 5m

    - name: baseline_traffic
      kind: profiling
      type: concurrency
      concurrency: 32
      duration: 30m

    - name: cancellation_stress
      kind: profiling
      type: concurrency
      concurrency: 64
      duration: 5m
      cancellation: {rate: 50, delay: 0}