DOCSEARCHbyAlgolia
Submit Docs

Mooncake Documentation

repository·main·Indexed 26 days ago

https://github.com/kvcache-ai/mooncake

A KVCache-centric disaggregated architecture for large-scale LLM inference and training. Mooncake optimizes performance by separating prefill and decode workloads and utilizing distributed CPU, DRAM, and SSD resources as a unified KV cache pool. The documentation covers the mooncake-transfer-engine (v0.3.12.post1), KVCache storage benchmarks, FAST'25 workload traces, collective communication benchmarks via pgbench.py, and monitoring setup using Prometheus and Grafana for the mooncake_master service.

Tokens
191K
Snippets
395
Records
812
Agent score
89%

What's inside Mooncake

  1. Overview of Mooncake Architecture

    main

    Mooncake is a KVCache-centric disaggregated serving platform designed for LLM serving. It separates prefill and decode clusters and utilizes underutilized CPU, DRAM, and SSD resources within GPU clusters to create a disaggregated KVCache pool.

    Key components include:

    • KVCache-centric Scheduler: Balances throughput and latency-related Service Level Objectives (SLOs). It uses prediction-based early rejection in overloaded scenarios to prevent wasting computation on requests that cannot meet SLOs.
    • Transfer Engine (TE): The central component for high-performance data transfer (e.g., RDMA-based P2P weight transfer).
    • Mooncake Store: A distributed KVCache engine built on top of the Transfer Engine.

  2. Overview of Mooncake Architecture

    main

    Mooncake is a multi-level cache pool designed to improve inference efficiency for Large Language Models (LLMs), particularly in environments with slow object storage. It leverages (GPUDirect) RDMA technology to enable zero-copy data transfers directly between the DRAM/VRAM of the initiator and the receiver, maximizing the utilization of multi-NIC resources on a single machine.

    Key capabilities include:

    • Object-level storage services: Provides Get, Put, List, and Del operations.
    • Flexible replication: Supports multiple replicas at the cache layer with slice-level distribution and best-effort allocation for a lightweight design.
    • Atomic writes: Guarantees that a Get operation will always read a complete object from a specific Put, though it may not necessarily be the most recent version.
    • Parallel I/O: Supports striping and parallel I/O for large objects to aggregate bandwidth across multiple network cards.
    • Configurable persistence: Supports three modes for flushing to slow object storage: Eager, Lazy, and None, allowing users to balance persistence requirements against performance.
    • Dynamic resource management: Supports adding or removing cache resources dynamically.

  3. Overview of TENT (Transfer Engine NEXT)

    main

    TENT (Transfer Engine NEXT) is a runtime designed for point-to-point data movement in heterogeneous AI clusters. It is the successor to the classic Mooncake Transfer Engine (TE).

    Unlike the classic TE, which required processes to bind to a single transport backend (like RDMA or NVLink), TENT abstracts transport details away from the application. It is designed to handle:

    • Heterogeneous interconnects: Automatically managing mixed environments (NVLink, RDMA, host memory).
    • Dynamic topology: Adapting to changing connectivity and link quality.
    • Partial failures: Handling slow or unavailable paths internally without surfacing errors to the application.

    Typical use cases include KVCache movement in LLM inference, data exchange between pipeline stages, and frequent model/parameter updates.

  4. Overview of Mooncake C/C++ APIs

    main

    Mooncake provides three primary C/C++ API modules for different layers of the system:

    1. Transfer Engine C++ API: Uses the TransferEngine class to handle memory registration, batch transfers, segment management, and RDMA transport.
    2. TENT C++ API: Uses mooncake::tent::TransferEngine, a next-generation transfer engine featuring automatic transport selection and fault tolerance.
    3. Mooncake Store Client C++ API: Uses the Client class to perform data operations including Put, Get, Remove, and Replicate, and provides the BufferAllocatorBase interface.

  5. Overview of Mooncake components

    main

    Mooncake is a KVCache-centric disaggregated architecture for LLM inference and training. It consists of three primary components:

    1. Transfer Engine (TE): A high-performance data transfer framework providing a unified interface for batched data movement across diverse storage, networks (TCP, RDMA, AWS EFA, NVMe-oF, etc.), and accelerators (CUDA, MUSA, HIP, Ascend, etc.).
    2. Mooncake Store: A distributed KV cache and model-weight management engine built on the Transfer Engine. It supports multi-tier caching (DRAM and SSD/NVMe), large-object striping, and programmatic object management (replica counts, pinning, etc.).
    3. Mooncake EP & PG:
      • Mooncake EP: Provides fault-tolerant expert-parallel dispatch and combine operations with active_ranks awareness.
      • Mooncake PG: A PyTorch distributed process-group backend that enables collective communication with failure detection and rank recovery.

  6. Overview of SGLang HiCache with Mooncake Backend

    main

    Mooncake acts as a high-performance L3 storage backend for SGLang HiCache. This integration allows for distributed KV cache storage across multiple servers using RDMA-accelerated data transfer.

    When an L1/L2 cache miss occurs, HiCache fetches the required KV cache from Mooncake's distributed memory pool. This setup provides:

    • Scalable Capacity: Aggregates cluster memory into large pools.
    • Cache Sharing: All SGLang instances in a cluster can share KV caches.
    • RDMA Acceleration: Low-latency, high-bandwidth transfer via direct memory access.
    • Zero Copy: Direct transfer between L2 and Mooncake to maximize throughput.

  7. Overview of Mooncake Python Modules

    main

    Mooncake provides several Python modules for interacting with its distributed systems:

    • Mooncake Store: A distributed KV cache storage client supporting put, get, remove, and replicate operations.
    • DataProto Structured Object Transfer: Helpers for storing and retrieving structured payloads similar to DataProto.
    • Transfer Engine: High-performance data transfer between nodes using RDMA or TCP.
    • EP Backend: An expert-parallel backend designed for deploying large Mixture-of-Experts (MoE) models.

  8. Overview of Heterogeneous Ascend Transport

    main

    Heterogeneous Ascend Transport is a high-performance data transfer library designed for heterogeneous inference scenarios. It manages KVCACHE transfers between different hardware architectures to optimize memory usage and throughput.

    Key Use Case:

    • 910B Devices: Perform PREFILL operations (inference computation).
    • H20 Devices: Perform DECODE operations (data decoding).
    • Transport Layer: Manages data exchange between 910B NPU memory and H20 GPU memory.

    Note: The current version only supports the WRITE semantic. READ semantic support is planned for future releases.

  9. Overview of Mooncake Store

    main

    Mooncake Store is a high-performance distributed KV Cache storage engine specifically designed for LLM inference scenarios. Unlike traditional cache systems (e.g., Redis or Memcached) where keys are derived from values via hashing, Mooncake Store is a dedicated storage engine for KV Cache, supporting object-level operations and version management requirements.

    Key features include:

    • Object-level operations: Simple API for Put, Get, and Remove.
    • Multi-replica support: Reduces hotspot pressure by placing object slices in different segments.
    • Strong consistency: Get operations always return complete and correct data; objects are immutable once Put until deleted.
    • Zero-copy & High bandwidth: Leverages the Transfer Engine for striping and parallel I/O to utilize multi-NIC aggregate bandwidth.
    • Dynamic scaling: Supports adding/removing nodes to handle load changes.
    • Fault tolerance: System remains operational as long as at least one master and one client are running.
    • Tiered caching: Supports offloading RAM cache to SSD to balance cost and performance.

  10. Overview of the Transfer Engine C++ API

    main

    The Transfer Engine C++ API is defined in mooncake-transfer-engine/include/transfer_engine.h. The API is divided into two categories:

    1. Core APIs: The minimal set of functions required to move data. This includes initializing the engine, registering memory, opening segments, submitting transfers, and querying transfer status.
    2. Advanced APIs: Optional functions for transport control, notifications, metadata/cache maintenance, topology discovery, and debugging.

  11. Overview of Mooncake SSD Offload

    main

    Mooncake Store supports offloading KV cache objects from distributed memory to local SSD. This allows extending effective cache capacity beyond DRAM limits at a lower cost.

    Key features:

    • Transparency: Offloading is handled by a background subsystem within the real client process. It is transparent to the application. A Put that would be evicted from memory is persisted to disk, and a Get that finds no memory replica automatically falls back to reading from SSD.
    • Performance: Uses zero-copy RDMA-based memory transfers to preserve performance characteristics of the hot path.