Envoy AI Gateway Documentation

Envoy AI Gateway provides inference optimization to improve the speed and reliability of AI/LLM tasks. It achieves this through intelligent routing and load balancing across backend endpoints.

Key capabilities include:

• Intelligent Endpoint Selection: Routing requests to the most suitable endpoints based on real-time metrics and availability.
• Dynamic Load Balancing: Distributing workloads across multiple backend instances to optimize resource utilization.
• Seamless Integration: Compatibility with both standard HTTPRoute and the enhanced AIGatewayRoute configurations.
• Extensible Architecture: Support for custom Endpoint Picker Providers (EPP), allowing developers to implement domain-specific routing logic.

The Envoy AI Gateway acts as a proxy for routing and managing LLM/AI traffic. It accepts requests using two primary API specifications:

1. OpenAI-compatible API: The gateway maintains compatibility with the OpenAI API specification, though it currently supports a subset of the full OpenAI API.
2. Anthropic-compatible API: The gateway supports Anthropic's API format for routing requests to various AI providers.

By using these compatible interfaces, you can route traffic intended for OpenAI or Anthropic to different underlying AI providers through the Envoy AI Gateway.

Envoy AI Gateway is an AI gateway built on CNCF's Envoy Gateway designed to simplify enterprise AI adoption. It provides a unified interface for interacting with various Large Language Model (LLM) providers and includes built-in management features for security and cost control.

Core Capabilities:

• Unified API: Provides a single integration point for multiple LLM providers (currently supporting AWS Bedrock and OpenAI).
• Upstream Authorization: Simplifies authentication processes across different AI providers.
• Usage Rate Limiting: Controls costs and operational efficiency by implementing rate limiting based on word tokens.

The Envoy AI Gateway provides support for the Model Context Protocol, acting as a transparent proxy between MCP clients (such as Claude or Goose) and backend MCP servers.

Key capabilities include:

• Aggregation: Combine multiple MCP servers into a single unified endpoint.
• Security: Apply OAuth authentication, fine-grained tool access control, and upstream API key injection.
• Tool Filtering: Control which specific tools are exposed to the client.
• Networking: Utilize Envoy's standard features like load balancing, rate limiting, circuit breaking, and observability (OpenTelemetry/Prometheus).
• Spec Compliance: Full support for the June 2025 MCP specification, including tool calls, notifications, prompts, resources, and bi-directional requests.

Envoy AI Gateway is designed to facilitate seamless communication between application clients and multiple Generative AI (GenAI) services by leveraging Envoy Gateway and Envoy Proxy.

Its primary objectives are:

• Enable GenAI traffic handling with Envoy: Provides control plane extensions to the Envoy Gateway API to define routing rules specifically for GenAI services.
• Easy Setup: Simplifies the management of traffic to and from GenAI services through preset Envoy Gateway configurations, allowing platform engineers to provide a gateway solution that application developers can use without deep infrastructure knowledge.

The project is an additive layer designed to expand Envoy use cases without disrupting existing Envoy deployment or control patterns.

Envoy AI Gateway v0.3 introduces several key capabilities for production-ready AI infrastructure:

• Endpoint Picker Provider: Enables intelligent AI load balancing.
• AIGatewayRoute with InferencePool: Allows for advanced routing configurations using inference pools.
• Google Vertex AI Support: Provides enterprise-scale AI integration via Google Vertex AI.
• Comprehensive AI Observability: Enhanced monitoring and visibility for AI workloads.
• Model Name Virtualization: Provides an abstraction layer for model names, allowing for greater flexibility in switching or upgrading models without changing client code.
• Unified APIs: Supports both LLM and non-LLM APIs through a unified interface.

Envoy AI Gateway provides specialized observability for AI and LLM workloads by extending the foundational observability of Envoy Gateway. The key features include:

• GenAI Metrics: Prometheus metrics that follow OpenTelemetry Gen AI semantic conventions. These allow you to monitor token usage, latency, and model performance.
• GenAI Tracing: OpenTelemetry integration using OpenInference semantic conventions, designed for LLM request tracing and evaluation.
• Access Logs with AI/LLM metadata: The ability to include AI-specific metadata (such as model name and token usage) directly in Envoy Access Logs.
• Gateway Configuration: Per-gateway configuration for the external processor container, which includes setting environment variables for tracing and defining resource requirements.

The Envoy AI Gateway MVP focuses on three core capabilities designed to address the unique challenges of LLM traffic compared to traditional API traffic:

1. Usage Limiting: Controls LLM usage based on word tokens rather than traditional request counts. This accounts for the computational complexity and cost associated with tokenization in LLM requests and responses.
2. Unified API: Simplifies client integration by providing a consistent interface for interacting with multiple different LLM providers.
3. Upstream Authorization: Provides a mechanism to configure and manage Authorization across multiple upstream LLM providers.

The aigw CLI is a command-line interface used to manage and interact with the Envoy AI Gateway. It currently supports two primary modes of operation:

1. Run: Executes the Envoy AI Gateway locally as a standalone proxy. This mode uses a provided configuration file and does not require external dependencies like Docker or Kubernetes.
2. Translate: Converts an Envoy AI Gateway configuration file into an Envoy Gateway configuration file format.

:::warning The CLI is experimental and currently under active development. :::

Envoy AI Gateway supports multiple LLM providers through a unified interface. You can manage connectivity, access provider-specific features, and use advanced capabilities like prompt caching.

Key areas for integration include:

• Provider Support: View the full list of supported AI/LLM service providers and their specific configuration requirements.
• Supported Endpoints: Review the available API endpoints and operations provided across different backend providers.
• Vendor-Specific Fields: Access unique provider capabilities by passing backend-specific parameters within OpenAI-compatible requests.
• Prompt Caching: Implement provider-agnostic prompt caching using the unified cache_control API.
• Connectivity: Establish fundamental connections between the gateway and your chosen AI providers.