KTransformers: CPU-GPU Heterogeneous Inference Framework

KTransformers is a research project designed for efficient inference and fine-tuning of Large Language Models (LLMs) using CPU-GPU heterogeneous computing. It optimizes performance for large Mixture-of-Experts (MoE) models by leveraging hybrid hardware resources.

The project provides two primary user-facing capabilities:

1. Inference: High-performance serving via kt-kernel using CPU-optimized kernel operations.
2. SFT (Supervised Fine-Tuning): Integration with LLaMA-Factory for fine-tuning ultra-large MoE models on limited GPU memory.

The hardware requirements differ significantly between the training (SFT) phase and the inference (serving) phase.

Training (LoRA SFT)

• GPU: 4 * NVIDIA RTX 4090 24GB (or equivalent with at least 48GB total VRAM).
• CPU: x86 CPU with AMX support.
• RAM: At least 2TB system memory (Swap can be used if insufficient).

Inference (LoRA Adapter + Original Model)

• GPU: 2 * NVIDIA RTX 4090 24GB (or equivalent with at least 48GB total VRAM).
• CPU: x86 CPU with AVX512F support (e.g., Intel Sapphire Rapids).
• RAM: At least 600GB system memory.
• Storage: ~600GB for model weights (native INT4 weight).

KTransformers provides a method to reduce the inference barrier for ultra-long context sequences (e.g., 128K to 1M tokens) on heterogeneous CPU/GPU systems. It leverages the sparsity of attention to achieve significant speedups compared to standard implementations like llama.cpp.

Key Performance Benchmarks (Example: InternLM2.5-7B-Chat-1M):

• 128K Context: Up to 7.1x faster than llama.cpp with 100% accuracy on 'needle in haystack' and 'passkey' tests. On kvretrieval datasets, it can achieve a 6.22x speedup while improving accuracy (e.g., 21.2 -> 24.4).
• 1M Context: On a single 24GB GPU, it can achieve ~16 tokens/s (nearly 10x faster than llama.cpp) and can improve 'needle in haystack' scores (e.g., 89.31 -> 92.88).

KT-Kernel enables CPU-GPU heterogeneous inference by running "hot" experts on the GPU and "cold" experts on the CPU. This allows for optimal resource utilization by leveraging both GPU VRAM and CPU memory/compute. To use this, you must integrate KT-Kernel with the kvcache-ai fork of SGLang.

python -m sglang.launch_server \
  [your normal SGLang parameters...] \
  --kt-method AMXINT8 \
  --kt-weight-path /path/to/cpu-weights \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 32 \
  --kt-max-deferred-experts-per-token 2

The KTransformers Docker image includes support for multiple CPU instruction sets: AMX (Intel Sapphire Rapids+), AVX512, and AVX2. While the runtime performs automatic detection, you can manually override the variant using the KT_KERNEL_CPU_VARIANT environment variable.

To force a specific variant (e.g., avx2) and debug the loading process, use:

export KT_KERNEL_CPU_VARIANT=avx2
export KT_KERNEL_DEBUG=1
python your_script.py

When using the merged adapter workflow, the model field in your OpenAI-compatible API requests follows a specific naming convention based on the --served-model-name and the name assigned in --lora-paths.

If you launch with: --served-model-name qwen3.5-kt-ft and --lora-paths qwen35b_lora=/path/to/merged_adapter

Then:

• model=qwen3.5-kt-ft results in: Base Model + KT Expert LoRA
• model=qwen3.5-kt-ft:qwen35b_lora results in: Base Model + KT Expert LoRA + SGLang Non-expert LoRA

The suffix after the colon (:) must exactly match the name provided on the left side of the --lora-paths assignment.

--served-model-name qwen3.5-kt-ft
--lora-paths qwen35b_lora=/path/to/merged_adapter

Request model names:
model=qwen3.5-kt-ft
model=qwen3.5-kt-ft:qwen35b_lora

KTransformers provides four strategies for distributing MoE experts between CPU and GPU via the --kt-expert-placement-strategy flag:

KTransformers has evolved through several versions, each introducing specific optimizations for DeepSeek models:

V0.2.1 Improvements

• Longer Context: Increased context support from 4K to 8K for 24GB VRAM setups.
• Triton MLA Kernel: Integrated the highly efficient Triton MLA Kernel (from sglang) to enable longer context and faster prefill/decode speeds.
• Speed: Approximately 15% speedup in prefill/decode over V0.2.

V0.3-Preview Optimizations

• Intel AMX Acceleration: Uses the Intel AMX instruction set and a cache-friendly memory layout for massive prefill speedups.
• Selective Expert Activation: Implements an expert selection strategy that selects fewer experts based on offline profiling. This reduces computational load without significantly impacting output quality.
• Performance: Prefill is up to 3.45x faster than V0.2.