DOCSEARCHbyAlgolia
Submit Docs

Gateway API Inference Extension

repository·main·Indexed 20 days ago

https://github.com/kubernetes-sigs/gateway-api-inference-extension

An extension to the Gateway API that optimizes self-hosting Generative AI models on Kubernetes by providing inference-aware routing and scheduling capabilities to compatible proxies.

Tokens
32.4K
Snippets
65
Records
132
Agent score
67%

What's inside gateway-api-inference-extension

  1. What is Lightweight Endpoint Picker (LWEPP)?

    main

    LWEPP is a minimal, lightweight reference implementation of the Endpoint Picker (EPP). It is designed to integrate with Envoy via the External Processing (ext_proc) protocol to manage request routing and response reporting.

    Its primary responsibilities include:

    • Envoy Integration: Using the ext_proc protocol to intercept request headers (to set target endpoint headers for Envoy routing) and response headers (to report which endpoint served the request).
    • Simple Load Balancing: Implementing basic round-robin load balancing across the available pods in a target pool.

  2. What is the Gateway API Inference Extension?

    main

    The Gateway API Inference Extension optimizes the self-hosting of Generative Models (such as LLMs) on Kubernetes. It transforms a standard gateway that supports both the [Gateway API] and Envoy's [External Processing (ext-proc)] into an Inference Gateway (IGW).

    An Inference Gateway provides:

    • Optimized Routing: Uses extensible scheduling algorithms that are aware of KV-cache and request costs to improve tail latency and throughput.
    • Kubernetes-native APIs: Declarative APIs for routing client model names to specific LoRA adapters, controlling incremental rollouts, A/B traffic splitting, and blue-green upgrades.
    • Observability: End-to-end monitoring of service objective attainment.
    • Operational Guardrails: Allows platform teams to safely serve multiple GenAI workloads on shared foundation model server pools, maximizing accelerator utilization.

  3. Overview of Gateway API Inference Extension

    main

    The Gateway API Inference Extension introduces two primary Custom Resource Definition (CRD) objects to manage machine learning inference workloads within a Kubernetes cluster: InferencePool and InferenceModel.

    • InferencePool: Represents a logical grouping of compute resources. It is typically managed by an Inference Platform Admin.
    • InferenceModel: Defines the serving objectives for a specific model or LoRA adapter. It is typically managed by an Inference Workload Owner.

    Note: This specific proposal is marked as Implemented/Obsolete. For the most current API specifications, refer to the /api/ and /apix/ directories in the repository or check the specific evolution proposals for InferencePool and InferenceModel.

  4. What is an InferencePool?

    main

    An InferencePool is a resource that defines a group of Pods (containers) dedicated to serving specific AI models. All Pods within a single pool share the same:

    • Compute configuration
    • Accelerator type
    • Base language model
    • Model server

    This abstraction allows Platform Admins to manage AI model serving resources centrally.

    Key Relationships

    • Endpoint Picker (EPP): An InferencePool is typically bundled with an Endpoint Picker extension. The EPP tracks metrics (like KV-cache utilization, queue length, or active LoRA adapters) to route requests to the optimal replica. While an EPP can only be associated with one InferencePool, an HTTPRoute can reference multiple InferencePools as backendRefs.
    • Model Server Protocol: To participate in an InferencePool, Pods must support the model server protocol so the Endpoint Picker can retrieve the metrics necessary for intelligent routing.
    • Endpoint Picker Reference: As of v1.5.0, the endpointPickerRef field is optional. If omitted, the pool can be used without a user-managed Endpoint Picker, but you should verify if your specific Inference Gateway implementation supports this.

  5. Overview of the Gateway API Scheduler

    main

    The Gateway API Scheduler is a subsystem designed to optimize request mapping to backend endpoints in an inference gateway. Unlike random load balancing, the scheduler uses a dynamic capacity model and anticipated request costs to achieve higher accelerator utilization and more predictable latency.

    It optimizes across several dimensions, including:

    • Request Characteristics: Prompt length and anticipated output length.
    • Backend State: Current traffic distribution and available backend KV-cache.
    • Performance & Topology: Heterogeneous accelerator performance, backend topology (e.g., prefill disaggregation), and model server tuning.
    • Routing Strategies: Anticipated savings from prefix cache-aware routing and workload latency objectives.

  6. What is verified in Conformance testing?

    main

    Conformance testing is divided into three distinct layers to ensure the entire inference ecosystem works together correctly:

    1. Gateway API Implementations

    Tests verify that the Gateway implementation:

    • Supports InferencePool as a backend type.
    • Forwards requests to the configured extension for an InferencePool according to the project specification.
    • Honors the routing guidance provided by the extension.
    • Behaves appropriately when an extension is missing or fails to respond.

    2. Inference Routing Extensions

    Tests verify that the extensions:

    • Accept requests matching the project's specified protocol.
    • Respond with routing guidance that matches the project's specified protocol.

    3. Model Server Frameworks

    Tests verify that the frameworks:

    • Serve the expected set of metrics using the specific format and path defined by this project.

  7. How BBR plugin execution chains work

    main

    The BBR framework uses a plugin architecture to avoid monolithic routing logic. Plugins can be organized into a PluginsChain to define an ordered execution sequence.

    • Request vs. Response Chains: You can configure RequestPluginsChain and ResponsePluginsChain to handle specific stages of the request lifecycle.
    • Default Behavior: If no specific configuration is provided, the system automatically configures default PluginsChain instances. The default plugin implementation sets the X-Gateway-Model-Name header based on the body but does not mutate the body.
    • Performance Optimization: To avoid redundant parsing, if any plugin in a chain implements the NeedsFullParsing interface, the body is parsed exactly once into a shared struct and passed to all subsequent plugins in the chain.
    • Mutation Safety: While plugins receive a shared parsed struct for read-only access, any plugin that needs to mutate the body must work on its own copy and return the mutated bytes separately.

  8. Understand the InferencePoolImport resource

    main

    The InferencePoolImport is a cluster-local, controller-managed resource used to represent an InferencePool that has been exported from another cluster.

    Key characteristics:

    • Not User-Authored: It is managed by a controller, not created directly by users. The status field reflects the effective import state.
    • Relationship Mapping: It maps an exported InferencePool to its original exporting controller and cluster name.
    • Namespace Sameness: It maintains the same name and namespace as the exported InferencePool to avoid unnecessary indirection.
    • Gateway API Integration: In a Gateway API context, an HTTPRoute can reference an InferencePoolImport to route matching requests to the endpoints of the remote, backing InferencePools.
    • Availability: Inference Platform Owners can reference an InferencePoolImport even if the local cluster does not currently host a local InferencePool.

  9. Manage InferencePool status using the shared ownership model

    main

    The InferencePool resource follows a shared ownership model similar to Kubernetes Services. This allows multiple controllers to manage the same InferencePool concurrently, enabling a single pool to be used as a backendRef by multiple HTTPRoutes attached to different Gateways managed by different implementations.

    Implementation Rules

    • DO NOT use labels, ownerReferences, or any other mechanism to claim exclusive ownership of an InferencePool.
    • ASSUME that other controllers may be reconciling the same InferencePool at any time.
    • TRACK your relationship to the InferencePool independently through your own status entries rather than through resource ownership.

  10. Implementation expectations for Gateway controllers

    main

    This project does not provide a default controller implementation. Its primary scope is to define the API extension model, provide conformance tests, and maintain documentation.

    If you are developing or using a Gateway controller, you must either:

    1. Implement your own extension logic.
    2. Use an existing extension implementation (such as the reference EPP implementation now located in the llm-d-router repository).

  11. Configure an InferencePool

    main

    An InferencePool is a logical grouping of compute (typically model server Pods) used to serve large models. It acts similarly to a Kubernetes Service but includes specialized routing for inference workloads.

    Key Responsibilities:

    • Enforcing fair resource consumption across competing workloads.
    • Efficiently routing requests across shared compute.

    Key Constraints:

    • It does not manage the lifecycle or deployment of Pods within the pool.
    • It does not enforce which adapters or base models are available on the Pods.
    • Any Pod joining the pool must support the project's protocol to allow intelligent routing.
    • An InferencePool requires at least one InferenceModel to be subscribed to it; otherwise, traffic will be rejected.

    Spec Fields:

    • selector: A map of labels used to identify the model server pods (must be in the same namespace).
    • targetPortNumber: The port used to access the selected model servers (1-65535).
    • endpointPickerConfig: Configuration for the extension service that picks endpoints for requests.
    apiVersion: inference.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: base-model-pool
spec:
  selector:
    app: llm-server
  targetPortNumber: 8080
  extensionRef:
    name: infra-backend-v1-app

  12. Understand the roles in the Inference Extension ecosystem

    main

    The Gateway API Inference Extension is designed around two primary personas to separate infrastructure management from workload management:

    Inference Platform Admin

    Responsible for the underlying infrastructure and operational stability. Key tasks include:

    • Managing hardware resources.
    • Deploying and maintaining Model Servers.
    • Managing Base Models.
    • Handling resource allocation for workloads.
    • Configuring Gateways.

    Inference Workload Owner

    Responsible for the lifecycle and configuration of specific Generative AI (LLM) workloads. Key tasks include:

    • Defining workload priority.
    • Managing fine-tunes, such as LoRA Adapters.
    • Managing System Prompts and Prompt Caches.
    • Orchestrating the rollout of adapters.