DOCSEARCHbyAlgolia
Submit Docs

Kthena Documentation

website·Indexed 35 days ago

https://kthena.volcano.sh

Documentation for Kthena, a platform for managing AI inference workloads. It features the Kthena Router for unified LLM access, Kthena Runtime for standardized Prometheus metrics, and a CLI for resource management. Key capabilities include prefill-decode disaggregation (supporting vLLM and SGLang), multi-node inference, KV-cache aware routing, session boost for multi-turn conversations, and Kubernetes Gateway API support.

Tokens
276.6K
Snippets
1.1K
Records
1.9K
Agent score
54%

What's inside Kthena

  1. Overview of Kthena Router

    v0.2.0
    Kthena Router is a standalone binary designed to provide unified access to Large Language Models (LLMs), supporting both public AI service providers (e.g., OpenAI, DeepSeek, HuggingFace, Google Gemini) and privately deployed models within a cluster. It is designed to be lightweight and can either integrate with existing gateway infrastructure (like Nginx or Envoy) or serve as a direct traffic entry point. For private deployments, it supports mainstream inference frameworks such as vLLM and SGLang, using real-time metrics (KV cache utilization, LoRA status) to make intelligent routing decisions that improve throughput and reduce latency.

  2. Overview of Kthena Token-Based Rate Limiting

    Kthena Router implements token-based rate limiting instead of traditional request-count limiting to account for the varying computational costs of AI inference. Limits are applied to input and output tokens over a specific time window (second, minute, hour, day, or month). Rate limiting is configured within the ModelRoute Custom Resource (CR). If a limit is exceeded, the router returns an HTTP 429 Too Many Requests error.

  3. Overview of Prefill-Decode Disaggregation for Huawei Ascend NPUs

    v0.3.0

    Kthena supports prefill-decode disaggregation to optimize LLM inference on Huawei Ascend NPUs. This architecture separates the workload into three primary components:

    1. Prefill Service: Processes input tokens and generates the initial key-value (KV) cache. It is compute-intensive and optimized for NPU tensor operations.
    2. Decode Service: Generates output tokens using the KV cache from the prefill phase. It is memory-intensive and optimized for sequential processing and NPU memory architecture.
    3. Communication Layer: Manages data transfer (KV cache) between prefill and decode services using the Huawei Collective Communication Library (HCCL) for high-speed NPU-to-NPU communication.

    Data Flow: Client Request $\rightarrow$ Prefill Service (KV cache generation) $\rightarrow$ Cache Transfer (via HCCL/Mooncake) $\rightarrow$ Decode Service (Token generation) $\rightarrow$ Client Response.

  4. Overview of the ModelServing plugin framework

    v0.3.0
    The ModelServing plugin framework allows for the customization of Pods created by the ModelServing controller without modifying the core API. It is designed to handle cross-cutting concerns such as inference engine specifics (vLLM, TensorRT-LLM), accelerator types (GPU, NPU), networking (RDMA), and cluster infrastructure integrations. Plugins are opt-in; if the plugins field is not configured, behavior remains unchanged. Multiple plugins can be composed and are executed in the order they are listed.

  5. Overview of Kthena Token-Based Rate Limiting

    v0.2.0
    Kthena Router implements token-based rate limiting rather than request-based limiting to account for the varying computational costs of AI inference. Limits are applied to input and output tokens over specific time windows (second, minute, hour, day, or month). Rate limiting is configured within the ModelRoute Custom Resource (CR). If limits are exceeded, the router returns an HTTP 429 Too Many Requests error.

  6. Overview of Kthena Router Fairness Scheduling

    Kthena Router fairness scheduling prevents a single user from dominating a model's serving capacity during periods of contention. Instead of strict arrival order (FIFO), the router maintains a per-model fairness queue that prioritizes users with lower recent usage.

    Fairness is enforced per model; usage on one model does not affect priority on another.

    Key Scheduling Rules:

    • Different users: Users with lower recent usage receive higher priority.
    • Same user: Requests are processed FIFO by arrival time.
    • Same priority: Earlier requests are served first.

    Constraint: Fairness scheduling and Session Boost are mutually exclusive. You must enable either ENABLE_FAIRNESS_SCHEDULING=true or ENABLE_SESSION_BOOST=true, but not both.

  7. Overview of Kthena Router Fairness Scheduling

    Fairness scheduling prevents a single user from dominating a model's serving capacity during periods of contention. Instead of strict arrival order (FIFO), the router maintains a per-model fairness queue that prioritizes users with lower recent usage.

    Key behaviors:

    • Per-Model Enforcement: Fairness is tracked per model; usage on one model does not affect priority on another.
    • User Priority: Users with lower recent usage get higher priority. Requests from the same user remain FIFO.
    • Mutual Exclusivity: Fairness scheduling is mutually exclusive with Session Boost. You must enable either ENABLE_FAIRNESS_SCHEDULING=true or ENABLE_SESSION_BOOST=true, but not both.

  8. Overview of Prefill-Decode Disaggregation for Huawei Ascend NPUs

    Prefill-decode disaggregation in Kthena separates the LLM inference process into two specialized services to optimize NPU resource utilization on Huawei Ascend hardware:

    • Prefill Service: Processes input tokens and generates the initial key-value (KV) cache. It is batch-friendly and optimized for high compute throughput and NPU tensor operations.
    • Decode Service: Generates output tokens sequentially using the KV cache from the prefill phase. It is latency-sensitive and optimized for NPU memory architecture.
    • Communication Layer: Manages high-speed data transfer of the KV cache between services using the Huawei Collective Communication Library (HCCL) and NPU-optimized shared storage or networks.

    Data Flow: Client Request $\rightarrow$ Prefill Service (KV cache generation) $\rightarrow$ Cache Transfer $\rightarrow$ Decode Service (Sequential token generation) $\rightarrow$ Client Response.

  9. Overview of Kthena Router architecture

    v0.1.0

    Kthena Router enables intelligent forwarding of inference requests to backend models using two primary Custom Resources (CRs):

    1. ModelServer: Defines backend inference service instances. It uses workloadSelector.matchLabels to identify and select the pods that belong to the server (pods must match all specified labels).
    2. ModelRoute: Defines the routing rules based on request characteristics such as model name, LoRA adapters, HTTP headers, and weight distribution.

    Supported routing strategies include simple model-based forwarding, weighted distribution (for canary releases or A/B testing), header-based routing, and PD-Disaggregated (Prefill/Decode) routing.

  10. Overview of Kthena Runtime

    v0.2.0
    Kthena Runtime is a lightweight sidecar service designed to standardize Prometheus metrics from inference engines, provide LoRA adapter download/load/unload capabilities, and support model downloading from S3, OBS, PVC, and HuggingFace to a local path.

  11. Overview of Kthena Router

    v0.4.0
    Kthena Router is a standalone binary designed to provide unified access to Large Language Models (LLMs). It supports both public AI service providers (e.g., OpenAI, DeepSeek, HuggingFace, Google Gemini) and privately deployed models within a cluster. For private deployments, it integrates with mainstream inference frameworks like vLLM and SGLang, monitoring real-time metrics (KV cache utilization, loaded LoRA details) to make intelligent routing decisions that improve throughput and reduce latency.

  12. Overview of the kvcache-aware score plugin

    The kvcache-aware plugin is a score plugin for the Kthena Router scheduler designed to reduce recomputation in multi-pod vLLM deployments. It routes inference requests to pods most likely to have matching KV cache entries using token-block based matching.

    How it works:

    1. Tokenizes the incoming prompt.
    2. Divides the token sequence into fixed-size blocks and hashes each block.
    3. Looks up which pods have cached each token block via the configured coordination backend.
    4. Scores pods based on the number of consecutive matching blocks from the start of the prompt; pods with more matches are preferred.