DOCSEARCHbyAlgolia
Submit Docs

AIBrix Documentation

repository·main·Indexed 26 days ago

https://github.com/vllm-project/aibrix

An open-source, cloud-native solution for scalable GenAI inference infrastructure optimized for deploying and managing large language models (LLMs) on Kubernetes. Includes documentation for AIBrix Chat (a multi-modal chat portal), AIBrix Console (a management layer for models, deployments, and quotas), and a comprehensive benchmarking suite for analyzing metrics like TTFT and TPOT.

Tokens
182K
Snippets
447
Records
848
Agent score
90%

What's inside AIBrix

  1. Overview of vLLM Semantic Router

    main
    The vLLM Semantic Router acts as an intelligent intermediary between clients and a model fleet. It sits in front of your models and classifies incoming prompts by topic and intent using an embedding-based domain classifier or keyword scanning. It then transparently rewrites the request—replacing the model field and optionally injecting system prompts or reasoning modes—to route the request to the most suitable backend model. This allows clients to use a single virtual model name (e.g., "model": "MoM") while the router handles specialized routing to models like qwen3-8b (for STEM/reasoning) or llama3-8b-instruct (for general/business tasks).

  2. Overview of AIBrix key features

    main

    AIBrix provides several specialized features for GenAI inference infrastructure:

    • High-Density LoRA Management: Support for lightweight, low-rank adaptations.
    • LLM Gateway and Routing: Traffic management across multiple models and replicas.
    • LLM App-Tailored Autoscaler: Dynamic scaling based on real-time demand.
    • Unified AI Runtime: A sidecar for metric standardization, model downloading, and management.
    • Distributed Inference: Scalable architecture for large workloads across nodes.
    • Distributed KV Cache: High-capacity, cross-engine KV reuse.
    • Cost-efficient Heterogeneous Serving: Mixed GPU inference with SLO guarantees.
    • GPU Hardware Failure Detection: Proactive detection of hardware issues.

  3. Overview of AIBrix features

    main

    AIBrix is an open-source, cloud-native solution for building scalable GenAI inference infrastructure. It is designed to deploy, manage, and scale large language model (LLM) inference for enterprise needs.

    Key capabilities include:

    • LLM Gateway and Routing: Traffic management across multiple models and replicas.
    • High-Density LoRA Management: Support for lightweight, low-rank adaptations.
    • Distributed Inference: Scaling workloads across multiple nodes.
    • LLM App-Tailored Autoscaler: Dynamic resource scaling based on real-time demand.
    • Unified AI Runtime: A sidecar for metric standardization, model downloading, and management.
    • Heterogeneous-GPU Inference: SLO-driven inference using diverse GPU types.
    • GPU Hardware Failure Detection: Proactive monitoring of GPU health.
    • KVCache Offloading and Cross-Engine KV Reuse: High-performance framework for KV offloading and reuse.
    • Benchmark Tool: Performance and resource efficiency measurement.

  4. Overview of AIBrix KV Cache Offloading

    main

    AIBrix KV cache offloading framework provides functionalities for cross-engine KV reuse, specifically designed for high-performance inference environments. Key features include:

    • Tensor Parallelism Aware Management: Ensures participating engine instances (e.g., vLLM) align KV tensors fetched from external cache backends to maintain a consistent view during cache misses.
    • Embedded Cache w/ CPU Memory: Provides a small CPU memory-based cache embedded within the engine to reduce latency by avoiding frequent remote cache backend access.
    • Selective KV Cache Offloading: Allows fine-grained control over offloading strategies via an extensible eviction policy layer. This helps mitigate network bandwidth contention in low-end GPU instances and preserves PCIe bandwidth in RDMA-equipped environments.

    Supported built-in eviction policies include:

    • LRU
    • FIFO
    • S3FIFO

  5. Overview of Cache Package Components

    main

    The cache package provides centralized caching and routing capabilities for AIBrix LLM inference. Key components include:

    • Cache Store (cache.go): Maintains pod metadata, model-to-pod mappings, request tracking, and GPU profiles.
    • Event Manager (kv_event_manager.go): Synchronizes KV cache events between vLLM pods and the routing system via ZMQ.
    • Event Handler (kv_event_handler.go): Processes incoming block stored/removed events and maintains consistency.
    • Request Tracking: Includes Request Trace for latency/token metrics and Output Predictor for capacity planning based on historical token counts.
    • Informers (informers.go): Kubernetes informers that watch Pod lifecycle, ModelAdapter resources, and configuration updates.

  6. Overview of ModelClaim High-Density GPU Runtime Pools

    main

    ModelClaim is an experimental feature that allows multiple independently managed model engines to share a single warm GPU runtime Pod. Instead of a 1:1 mapping between a Kubernetes Deployment and a model, an operator creates a pool of topology-homogeneous GPU Pods, and users create ModelClaim objects to run specific models within those pools.

    Key Benefits:

    • Resource Efficiency: Multiple engines share the same GPU resources and Pod.
    • Elastic KV-Cache: The kvcached framework provides elastic memory across colocated engines.
    • Automatic Management: Supports optional pool policies for KV capacity redistribution and putting idle vLLM engines into sleep mode.

    Warning: This is an experimental feature. It currently supports only one engine replica per claim and requires a dedicated kvcached runtime image.

  7. Overview of KVCache Offloading

    main

    AIBrix KVCache Offloading is a framework that can be used as a standalone component without requiring the full AIBrix stack installation. It provides mechanisms for managing KVCache across different tiers (L1 and L2).

    Compatibility & Support:

    • vLLM Connectors: Supports both vLLM V0 and V1 connectors (since v0.4.0).
    • Engines: Supports both vLLM and SGLang (since v0.5.0).
    • Attention Kernels: Currently, only FlashAttention and XFormers are supported.

  9. Understand the AIBrix Architecture

    main

    AIBrix is divided into two primary layers: the Control Plane and the Data Plane.

    • The Control Plane manages model metadata registration, autoscaling, model adapter registration, and policy enforcement.
    • The Data Plane provides the components for dispatching, scheduling, and serving inference requests to enable high-performance model execution.

  10. Understand AIBrix StormService Architecture

    main

    StormService is a specialized component for managing the lifecycle of inference containers in disaggregated architectures (like Prefill/Decode). It uses a three-layer architecture based on Custom Resource Definitions (CRDs):

    1. StormService: The top-level CRD that defines the service specification, tracks status (including Replicas), and manages RoleSets and update strategies.
    2. RoleSet: A collection of roles (e.g., Prefill or Decode) where each role serves a specific function.
    3. Pods: The actual containers executing the inference tasks within a role.

    Updates propagate from the StormService down to RoleSets, and then to individual roles.

  11. Understand the AIBrix Engine Runtime

    main

    The AIBrix Engine Runtime (AIRuntime) is a unified management layer that operates as a sidecar to inference containers. It acts as a bridge between the AIBrix Control Plane and inference engine pods.

    Key capabilities include:

    • Model Management: Provides a universal method for model downloading, loading, unloading, and local management.
    • Control Plane Integration: Enables components like the LoRA adapter controller, autoscaler, and cold start manager to interact dynamically with inference containers.
    • Engine Abstraction: Abstracts vendor-specific APIs (e.g., for different inference engines) into a consistent interface for adapter configuration and performance monitoring.
    • Observability: Standardizes metrics across different inference engines for consistent monitoring and troubleshooting.

    Note: The AIRuntime is NOT an Istio sidecar. Data plane traffic does not pass through this runtime; it is strictly used for management and control plane interaction.

  12. Understand Redis Storage Versioning in AIBrix

    main

    AIBrix uses Redis sorted-set timestamp indexes to manage storage. The system supports multiple version schemas for how timestamps and keys are stored. The current latest version is v3.

    Version Specifications

    • v1: Stores positive Unix timestamps. Only oldest-first listing is available based on the stored score direction.
    • v2: Stores negative Unix timestamps. ZRANGE returns newest-first results because newer objects have smaller scores. Pagination and rank logic remain unchanged.
    • v3 (Latest):
      • Stores positive Unix timestamps as the canonical created_at score.
      • Supports both created_at_asc and created_at_desc ordering using ZRANGE/ZRANK (ascending) and ZREVRANGE/ZREVRANK (descending).
      • Uses slash-delimited schemas for batch jobs: batchjob/<id> (previously batchjob:<id>) and batchstatus_copies:<job_id>/<worker_id> (previously batchstatus_copies:<job_id>:<worker_id>).
      • Rebuilds hierarchical Redis indexes (parent:index and timestamps:{parent}) to support batch metastore listings.