candle-vllm

Candle-vLLM is a high-performance LLM inference and serving platform with the following capabilities:

• API Compatibility: Provides an OpenAI-compatible API service.
• Inference Optimizations: Supports PagedAttention, Continuous Batching, Prefix Caching, and CUDA Graphs.
• Quantization Support: Supports In-situ quantization (including Marlin format), GPTQ, AWQ, Marlin (4-bit), and hardware-specific FP8 (SM90+).
• KV Cache Compression: Features TurboQuant (turbo8, turbo4, turbo3) for high-ratio KV cache compression using native Flash Attention kernels.
• Multi-Device Support: Supports multi-GPU (multi-process and multi-thread tensor parallelism) and multi-node (TCP-based) inference.
• Platform Support: Works on CUDA (Linux) and Metal (macOS).
• Advanced Features: Supports Model Context Protocol (MCP), tool calling, speculative decoding (via --mtp), and chunked prefilling (default block size 8K).

candle-vllm supports the Model Context Protocol (MCP) and OpenAI-style tool calling. It follows the standard OpenAI flow: the server injects tool definitions into the prompt and parses model-emitted tool calls, but the client remains responsible for executing the tools and sending the results back.

The Tool Calling Workflow:

1. Configure: Set up MCP servers via CLI flags or an MCP config file.
2. Injection: candle-vllm loads tool definitions and injects them into the prompt.
3. Emission: The model emits a tool call, and the response finishes with finish_reason="tool_calls".
4. Execution: The client executes the tool locally.
5. Follow-up: The client sends the result back to the server as a message with role="tool" and the corresponding tool_call_id.

// 1. Request with tools
{
  "model": "default",
  "messages": [
    {"role": "user", "content": "List files in the current directory"}
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "list_files",
        "description": "List files in a directory",
        "parameters": {
          "type": "object",
          "properties": {
            "path": {"type": "string"}
          },
          "required": ["path"]
        }
      }
    }
  ]
}

// 2. Assistant response with tool_calls
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "tool_calls": [
          {
            "id": "call_123",
            "type": "function",
            "function": {
              "name": "list_files",
              "arguments": "{\"path\":\".\"}"
            }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ]
}

// 3. Client follow-up with tool result
{
  "role": "tool",
  "tool_call_id": "call_123",
  "content": "file1\nfile2\nfile3"
}

TurboQuant uses the Walsh-Hadamard transform to compress the KV cache, enabling higher throughput and longer context windows. It supports CUDA (SM70+) and Metal (Apple Silicon). Note that MLA models (e.g., DeepSeek, GLM4/GLM-5.2) will automatically fallback to standard KV cache due to layout incompatibility.

# Turbo4 (approx 3.7x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo4

# Turbo8 (approx 2.6x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo8

The candle-vllm CLI can be used to serve various model formats including FP8, FP4, GGUF, and Marlin-compatible GPTQ/AWQ. By default, it starts an OpenAI-compatible API service at http://localhost:2000. Adding the --ui-server flag starts a built-in ChatGPT-style Web UI.

Common Usage Scenarios

• FP8 Models: Use --m with the model ID.
• GGUF Models: Use --m for local files or directories. For HuggingFace repositories, use --m <repo> --f <file_or_subdir>.
• In-place Quantization (ISQ): Run unquantized models with the --isq flag.
• Marlin/AWQ/GPTQ: Use --w to point to local weights after conversion.
• Speculative Decoding: Use --mtp <n> for MTP draft tokens.

# FP8 model + Web UI
candle-vllm --m Qwen/Qwen3.6-27B-FP8 --ui-server

# GGUF model from HuggingFace (specific file)
candle-vllm --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# In-place quantization (ISQ)
candle-vllm --m Qwen/Qwen3.6-27B --isq q4k

# Speculative decoding (2 draft tokens)
candle-vllm --w /data/Qwen3.5-35B-A3B-FP8/ --mtp 2 --ui-server

Use the candle-vllm CLI to serve models. You can use HuggingFace Model IDs or local file paths.

Using HuggingFace Model IDs

Pass the model ID to the --m flag. You can also specify quantization formats with --f and device IDs with --d.

# Standard HuggingFace model
candle-vllm --m Qwen/Qwen3.6-27B-FP8 --ui-server

# Model with specific quantization and multiple GPUs
candle-vllm --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_S --d 0,1 --ui-server

# Model distributed across many GPUs
candle-vllm --m zai-org/GLM-5.2-FP8 --d 0,1,2,3,4,5,6,7 --ui-server

Using Local Model Paths

Supported local formats include safetensors directories, single/split-shard GGUF files, or directories containing GGUF files.

# Local safetensors directory
candle-vllm --d 0,1,2,3,4,5,6,7 --m /home/data/GLM-5.2-FP8/ --ui-server

# Local single GGUF file
candle-vllm --d 0,1 --m /home/data/model-Q4_K_M.gguf --ui-server

# Local directory containing GGUF files (auto-detected)
candle-vllm --d 0,1 --m /home/data/Qwen3.5-35B-A3B-GGUF/ --ui-server

Launching the Web UI

Add the --ui-server flag to launch a built-in ChatGPT-style Web UI.

Note on Ports: The UI server automatically uses the API port minus one. For example, if the API is running on port 2000, the UI will be available on port 1999.