vLLM Semantic Router Documentation

The ML-Based Model Selection E2E Profile demonstrates how to use pretrained machine learning models for intelligent model selection at runtime. This implementation follows concepts from the FusionFactory and Avengers-Pro research papers to optimize performance and efficiency by selecting the best model for a specific query.

This profile is production-ready and includes a full stack:

• Envoy Gateway: Gateway API implementation.
• Envoy AI Gateway: AI-specific Custom Resource Definitions (CRDs) and routing.
• gRPC ExtProc: The semantic router acting as an external processor.
• Custom AIGatewayRoute: Routes requests based on the x-selected-model header to a backend (e.g., a mock-llm).

TruthLens is a real-time hallucination detection and mitigation framework integrated into the vLLM Semantic Router. It operates at the inference gateway layer, providing a model-agnostic, centralized solution for detecting and addressing hallucinations (factually incorrect or ungrounded content).

TruthLens allows users to manage the "accuracy-latency-cost" triangle by selecting from three operational modes based on their specific requirements for cost and accuracy.

The E2E Test Framework is a comprehensive testing suite for Semantic Router designed to support multiple deployment profiles. It uses a separation of concerns architecture consisting of:

• Profiles: Define specific deployment environments and configurations (e.g., kubernetes, istio, production-stack).
• Test Cases: Reusable logic shared across different profiles.
• Framework: The core infrastructure for executing tests and generating reports.
• Stacks: Reusable deployment modules for common topologies like semantic-router/envoy/ai-gateway.
• Fixtures: Typed service sessions and API clients for common contract families.

The vLLM Semantic Router is an intent-aware routing layer designed for the vLLM inference engine. It optimizes LLM serving by classifying queries semantically and routing them to different inference paths (e.g., a 'fast path' for simple queries and a 'Chain-of-Thought' reasoning mode for complex queries).

Key architectural components include:

• Semantic Classification: Uses ModernBERT as a lightweight, standalone classifier to determine routing paths.
• Smart Routing: Directs queries to either fast-path inference or reasoning-enabled models based on intent.
• High-Performance Engine: Built in Rust using Hugging Face Candle for high concurrency and zero-copy inference.
• Cloud-Native Integration: Supports out-of-the-box integration with Kubernetes and Envoy via the ext_proc plugin.

The modelselection module provides machine learning algorithms to route queries to the most appropriate LLM based on query characteristics and historical performance.

Supported algorithms include:

• KNN (K-Nearest Neighbors): Uses quality-weighted voting among similar queries. Implemented in Rust via Linfa.
• KMeans: Cluster-based routing with quality-weighted assignment. Implemented in Rust via Linfa.
• SVM (Support Vector Machine): Decision boundaries with RBF kernel. Implemented in Rust via Linfa.
• MLP (Multi-Layer Perceptron): GPU-accelerated neural network routing using Candle (supports CUDA/Metal).

Fleet Sim is a fleet simulator designed to help plan GPU fleets before deployment. It is used for sizing homogeneous, heterogeneous, or disaggregated fleets against latency targets, comparing annualized costs across GPU choices and routing policies, and validating planning assumptions via simulation runs, trace replay, and what-if analysis.

Note: Fleet Sim is not part of the router's live request path, nor is it a runtime autoscaler, burst controller, or per-kernel profiler.

The data/ directory contains pre-processed Cumulative Distribution Function (CDF) files used for workload simulation in examples and the CLI. Each file is a JSON array of [token_length, cumulative_fraction] pairs representing the empirical CDF of total token counts (input + output) for a specific trace.

Available CDF files:

• azure_cdf.json: Azure LLM Inference Trace 2023 (28K prod requests; p90=4.2K tokens)
• lmsys_cdf.json: LMSYS-Chat-1M (Single-turn conversations)
• lmsys_multiturn_cdf.json: LMSYS-Chat-1M (multi-turn, accumulated context per turn)
• agent_heavy_cdf.json: Synthetic agent-heavy (SWE-bench 40% + BFCL 25% + RAG 35%)

Using semantic tool selection instead of loading all tool definitions upfront provides several key advantages for AI agents:

• Restores Usability at Scale: Prevents the catastrophic accuracy drop (often 79-100%) seen in models when tool catalogs grow beyond ~50-100 tools. It helps maintain high accuracy (e.g., ~94%) even as the catalog expands.
• Dramatic Token & Cost Reduction: Can reduce token consumption by up to 99.1%. For example, a catalog of 741 tools can be reduced from ~127K tokens to ~1K tokens per request.
• Eliminates Position Bias: Solves the "lost in the middle" effect where LLMs struggle to select tools located in the middle of a long context window.
• Enables Longer Conversations: By saving significant context space on tool definitions, more tokens are available for conversation history and tool responses.

The Signal-Decision Architecture is a replacement for traditional classification-based routing. Instead of routing based on a single MMLU domain category, it uses a multi-dimensional approach to capture complex user intent.

Key components include:

• Multi-Signal Extraction: Captures multiple dimensions of intent (e.g., urgency, security, intent type, complexity) simultaneously.
• Decision Engine: A flexible engine that combines extracted signals using AND/OR logic and priority-based selection.
• Plugin Chain: Orchestrates built-in intelligence for tasks like caching, security, and optimization.

This architecture allows for unlimited intelligent routing decisions beyond fixed categories, enabling enterprise-grade requirements like combining security filtering with specialized model routing.