InfiniStore Documentation

InfiniStore is a high-performance, open-source Key-Value (KV) store designed specifically for LLM (Large Language Model) inference clusters. It facilitates high-performance, low-latency KV cache transfer and reuse among inference nodes.

It supports two primary deployment scenarios:

1. Prefill-Decoding Disaggregation Clusters: Enables KV cache transfer and reuse between dedicated prefill nodes and decoding nodes.
2. Non-disaggregated Clusters: Acts as an extended KV cache pool (supplementing GPU and local CPU cache) and enables cross-node KV cache reuse when prefill and decoding workloads are mixed on the same nodes.

Currently, InfiniStore is integrated with vLLM via LMCache. Integration with SGLang and other engines is in progress.

InfiniStore is a high-performance system designed to support disaggregated LLM inference architectures. When used alongside vLLM, it extends the capabilities of a single-instance setup to a cluster-level architecture.

Key features include:

• Prefill-decoding architecture: Supports the disaggregation of prefill and decode phases to optimize computational and memory demands.
• Expanded Historical KV Cache: Provides a much larger pool for historical KV cache by utilizing DRAM and SSD, whereas vLLM's standard Automatic Prefix Cache (APC) is limited to GPU HBM.
• Cross-host KV cache: Enables a host to reuse historical KV cache stored on other hosts in the cluster.

InfiniStore utilizes a traditional key-value structure that supports variable-length keys. This allows developers to encode metadata such as model_id, request, and token hash directly into the keys.

To mitigate the high latency of RDMA memory registration, InfiniStore performs the following:

• Pre-registration: RDMA memory is pre-registered during the startup phase.
• Memory Pooling: Implements memory management using a memory pool.
• Management Algorithms: Supports jemalloc or bitmap algorithms, with bitmap being the default.

InfiniStore is typically deployed on the same server as vLLM, sharing local CPU and memory resources. The data flow for KV cache management follows these patterns:

Prefill Stage

vLLM writes to the kvcache layer by layer. This can be done via local GPU copy or RDMA. Writing layer-by-layer allows for parallelizing network communication with GPU computation, minimizing network overhead (measured at <1% in practice).

Decode Stage

To prevent network operations from blocking the GPU, a separate thread in vLLM is used to download the KV cache. Once the download is complete, this thread notifies the scheduler to begin the decoding process.

When submitting code changes via Pull Request, ensure your changes do not break existing features by running the following checks:

1. Run Unit Tests: pytest infinistore/test_infinistore.py
2. Run Pre-commit Checks: pre-commit run --all-files

InfiniStore can run on either GPU or CPU machines. The startup command depends on your network configuration (TCP/IP or RDMA).

1. Start the InfiniStore Server

For TCP/IP Network:

infinistore --service-port 12345

For RDMA (RoCE):

infinistore --service-port 12345 --dev-name mlx5_0 --link-type Ethernet

For RDMA (Infiniband):

infinistore --service-port 12345 --dev-name mlx5_0 --link-type IB

2. Run the InfiniStore Client

Example client implementations can be found in the repository at:

• infinistore/example/client.py
• infinistore/example/client_async.py
• infinistore/example/client_async_single.py

# Example for RDMA (RoCE)
infinistore --service-port 12345 --dev-name mlx5_0 --link-type Ethernet

To verify that InfiniStore is installed correctly, start a management port and run a self-test using curl.

1. Run the management command: infinistore --manage-port 8088
2. In a separate terminal, check the self-test endpoint: curl http://127.0.0.1:8088/selftest

For most users who only need to deploy and run InfiniStore without modifying its internal code, use the standard PIP installation:

pip install infinistore

To use InfiniStore for KV cache transfer and reuse within a vLLM cluster, you must ensure the following components are installed on all nodes in the cluster:

1. vLLM
2. LMCache
3. InfiniStore

Specific setup requirements may vary based on your unique vLLM cluster configuration.

For developers contributing to the codebase, install from source. You must first install the required system dependencies via apt, then install the package in editable mode.

Prerequisites (System Dependencies):

• libuv1-dev
• libflatbuffers-dev
• libspdlog-dev
• libfmt-dev
• ibverbs-utils
• libibverbs-dev
• libboost-dev
• libboost-stacktrace-dev

Installation Steps:

1. Install system dependencies.
2. Install the package using pip install --no-build-isolation -e ..
3. Install and set up pre-commit hooks.

apt install libuv1-dev
apt install libflatbuffers-dev
apt install libspdlog-dev libfmt-dev
apt install ibverbs-utils libibverbs-dev
apt install libboost-dev libboost-stacktrace-dev
pip install --no-build-isolation -e .
pip install pre-commit
pre-commit install

To prevent memory exhaustion, InfiniStore can run a background task that periodically calls evict_cache. To enable this, use the --enable-periodic-evict flag. You can tune the behavior using:

• --evict-interval: How often the task runs (seconds).
• --evict-min-threshold: The lower bound for eviction logic.
• --evict-max-threshold: The upper bound for eviction logic.

# Example: Enable eviction every 10 seconds with custom thresholds
python infinistore/server.py --enable-periodic-evict --evict-interval 10 --evict-min-threshold 0.5 --evict-max-threshold 0.7

You can run the InfiniStore service using the server.py entrypoint. The service starts a control plane (management API) and a data plane. It uses uvloop for high-performance asynchronous I/O and can be configured via several command-line arguments to manage memory allocation, InfiniBand (IB) settings, and cache eviction policies.

python infinistore/server.py --manage-port 18080 --service-port 22345 --prealloc-size 32 --enable-periodic-evict