KsanaLLM

KsanaLLM is a high-performance and easy-to-use inference engine designed for Large Language Model (LLM) inference and serving. It is optimized for high throughput and supports both NVIDIA GPUs and Huawei Ascend NPUs.

Key Performance Features

• Optimized Kernels: Utilizes high-performance operators from projects like vLLM, TensorRT-LLM, FasterTransformer, SGLang, and LightLLM.
• Memory Management: Implements PagedAttention for efficient KV cache management.
• Dynamic Batching: Fine-tuned task scheduling and memory footprint optimization.
• Advanced Features: Supports Prefix caching and DeepSeek-MTP.

Key Usability Features

• Model Compatibility: Seamlessly integrates with Hugging Face models (supports PyTorch and SafeTensor formats).
• Serving Capabilities: Supports high-throughput serving with multiple decoding algorithms (e.g., parallel sampling, beam search), streaming output, and an OpenAI-compatible API server.
• Parallelism: Supports tensor parallelism across multiple cards.

Supported Hardware

• NVIDIA GPUs: A10, A100, L40, L20, H20
• Huawei Ascend NPUs: 910B2C

ModelPerfBackend is a pure performance testing mode designed for the event-driven scheduler. Unlike the standard inference mode, it does not accept external RPC/HTTP requests. Instead, it follows a deterministic execution flow:

1. It reads a batch_config.json file to construct fixed batches of requests.
2. It executes multiple rounds of warmup (to stabilize performance) and stat (to collect metrics).
3. It collects scheduling and operator-level latency data.
4. It writes the results to a JSON file (specified by output_file in model_perf_config.json) and then automatically exits the process.

This mode is intended for performance profiling and simulator fitting rather than serving live traffic.

InferenceEngine::Start()
  └─ ModelPerfBackend::StartPerf()          # Blocks until all batches complete
       for each batch in batches:
         1. Construct InferRequest, submit via AddInferRequest
         2. ModelPerfBackend::Schedule()     # Multi-round (warmup + stat) scheduling
         3. UpdateWithGenerationResult()     # Collect metrics per round
         4. batch_waiter returns, move to next batch
      Write model_perf_result.json
  └─ InferenceEngine::Stop() → Process exits

Remote Cache Loading is an optimized process for loading prefix cache data from remote storage. The workflow follows a specific state machine and sequence of API calls to manage FA (Flash Attention) and SWA (Streaming Window Attention) blocks during loading.

Typical Workflow Sequence

1. Check Cache Status: Call GetRequestPrefixBlockNumberBeforeRemoteCacheLoading to determine the current cache hit range.
2. Allocate Resources: Call AllocateBlocksForRemoteCache to reserve FA and SWA blocks for the incoming remote data.
3. Perform Remote Load: The caller (Engine) performs the actual data transfer from remote storage using the allocated block IDs.
4. Confirm Results: Call ConfirmRemoteCacheLoadingResult to finalize the process. This step transfers successfully matched SWA blocks to their actual FA block positions and releases any unneeded blocks.
5. Resume Inference: Continue inference using the updated cached_blocks.

Request States during Loading

While a request is in the remote_loading state, certain inference-related interfaces are prohibited:

• AllocateRequestBlocks
• UpdateRequestTokens
• AppendFilledCachedBlock

Note that the state is tracked via is_remote_loading and is_remote_loading_block_allocated within the PrefixCachedRequest structure.

/* Sequence of operations for an Engine */
1. GetRequestPrefixBlockNumberBeforeRemoteCacheLoading()
2. AllocateBlocksForRemoteCache(fa_block_num, swa_block_num)
3. [Perform Remote Loading]
4. ConfirmRemoteCacheLoadingResult(matched_block_index, matched_remote_swa_block_indexs, reserved_block_num)
5. [Continue Inference]

When matching prefixes in the cache, a matched endpoint is only considered valid if it satisfies the Continuous Window Condition. Because SWA requires every token to see its previous W tokens, a single missing SWA block breaks the continuity.

The Rule: An endpoint is valid if and only if the last W_blocks (where W_blocks = ceil(W / B)) are all valid in the SWA cache.

Matching Process (Two Phases):

1. Phase 1 (Scan): Traverse the prefix tree. Use a swa_valid_in_window counter. Increment it for valid SWA blocks; reset it to 0 if an invalid block is encountered. The last_swa_valid_idx is updated only when the counter reaches sliding_window_blocks.
2. Phase 2 (Registration): Once the furthest valid endpoint is found, register the blocks in the range [win_start, last_swa_valid_idx) with the SWACacheManager by calling SetReclaimable(fa_block_id, false) and IncrementHitCount(fa_block_id).

When the SWA memory pool is exhausted and a new block must be allocated, SWACacheManager follows a specific priority and eviction strategy to preserve high-value prefix cache data.

Allocation Priority

1. Highest Priority: Allocate directly from the idle pool using the BlockAllocatorGroupInterface. If blocks are available here, no eviction occurs.
2. Lowest Priority: If the idle pool is empty, trigger eviction of reclaimable blocks where is_content_fixed=true.

Eviction Selection (LFU + Prefix Position)

To maximize the computational savings of the prefix cache, SWACacheManager selects the "lowest value" block for eviction using a composite sort on reclaimable, content-fixed blocks:

1. Primary Key: hit_count (Ascending) Blocks with fewer hits are considered less valuable and are evicted first (LFU strategy).
2. Secondary Key: prefix_fa_block_num (Ascending) If hit counts are equal, the block with the smaller prefix_fa_block_num (the number of preceding FA blocks) is evicted first. A smaller index means the block provides less computational savings when reused.

Note: Blocks where is_content_fixed=false are never candidates for eviction; they are either actively being written to or are directly freed.

Simple Router is a FastAPI-based coordination service for KsanaLLM deployments. It manages the relationship between Prefill Nodes (which process prompts and generate KV cache) and Decode Nodes (which generate subsequent tokens autoregressively).

Core Workflow

1. Node Registration: Nodes register via /RegisterNode and maintain status via /Heartbeat.
2. Pairing: The router tracks available prefill/decode pairs in a database. Any prefill node can be paired with any decode node (fully meshed topology).
3. Request Handling: When a client sends a request to the router's proxy endpoints (/generate, /v1/*, or /v2/*), the router:
   • Selects an available prefill/decode pair.
   • Generates a unique comm_id.
   • Forwards the request to both nodes simultaneously using custom headers (kv-comm-group-key, kv-comm-request-id) to coordinate KV cache transfer.
   • Merges the streaming responses (prefill's first token + decode's subsequent tokens) and returns them to the client.

Cluster Topology

A cluster typically consists of $N$ prefill nodes and $M$ decode nodes. The router provides load balancing and fault tolerance by dynamically selecting pairs based on node health and availability recorded in the node_info table.