vllm-mlx Documentation

vLLM-MLX is an Apple Silicon MLX backend for vLLM that provides native GPU acceleration for Text, Image, Video, and Audio inference on Mac. It integrates several specialized MLX frameworks to achieve high performance:

• MLX: Apple's ML framework utilizing unified memory and Metal kernels.
• mlx-lm: Optimized LLM inference featuring KV cache and quantization.
• mlx-vlm: Support for vision-language models (multimodal inference).
• mlx-audio: Text-to-Speech (TTS) and Speech-to-Text (STT) capabilities.
• mlx-embeddings: Text embeddings for RAG and semantic search.

The vllm-mlx CLI provides several primary commands for managing and benchmarking models:

• vllm-mlx serve: Starts an OpenAI-compatible API server.
• vllm-mlx model: Used to inspect, acquire, or convert model artifacts.
• vllm-mlx bench-serve: Benchmarks a running server using prompt sweeps or workload contracts.
• vllm-mlx-bench: Runs performance benchmarks.
• vllm-mlx-chat: Starts a Gradio-based chat interface.

| Command | Description |
|---------|-------------|
| `vllm-mlx serve` | Start OpenAI-compatible server |
| `vllm-mlx model` | Inspect, acquire, or convert model artifacts |
| `vllm-mlx bench-serve` | Benchmark a running server with prompt sweeps or workload contracts |
| `vllm-mlx-bench` | Run performance benchmarks |
| `vllm-mlx-chat` | Start Gradio chat interface |

vllm-mlx supports several categories of models via specialized MLX backends:

• Language Models (via mlx-lm): Includes Llama (3.x, 4.x), Mistral/Devstral, Qwen (2/3), DeepSeek (V3, R1), Gemma (2, 3, 4), Phi-3, Granite, and more.
• Multimodal Models (via mlx-vlm): Includes Qwen-VL, LLaVA, Idefics, Gemma 4 (vision + audio), PaliGemma, Pixtral, Molmo, Phi-3 Vision, and DeepSeek-VL.
• Embedding Models (via mlx-embeddings): Includes BERT, XLM-RoBERTa (e.g., multilingual-e5), and ModernBERT.
• Audio Models (via mlx-audio):
  • STT (Speech-to-Text): Whisper, Parakeet.
  • TTS (Text-to-Speech): Kokoro, Chatterbox.

All quantized models from the mlx-community on HuggingFace are compatible.

The vllm-mlx package provides three primary CLI tools:

• vllm-mlx serve: Starts an OpenAI-compatible API server.
• vllm-mlx-bench: Runs performance benchmarks.
• vllm-mlx-chat: Launches a Gradio-based chat interface.

The project implements a prefix cache to optimize performance for repeated multimodal requests. The cache key strategy is based on the following formula:

hash(image_content) + hash(prompt)

Cache Behavior Rules:

• Cache HIT: Occurs when you provide the same image and the same prompt as a previous request.
• Cache MISS: Occurs if the image is different, or if the prompt is different, even if the image is the same.

Warm-up prompts are processed concurrently using asyncio.gather. This means $N$ entries in your JSON file will trigger $N$ concurrent prefills at startup, each allocating KV cache for its prompt length.

Best Practices:

• Recommended size: 1–3 entries. This typically covers the hot paths for most agent deployments.
• Memory Warning: A very large warm-prompts file can exhaust memory headroom at boot, especially on memory-tight models.
• Concurrency Control: Currently, there is no built-in cap for concurrency, but users can request a --warm-prompts-concurrency=N flag via GitHub issues for large workloads.

The video cache mechanism ensures that repeated requests with identical parameters avoid redundant processing. The cache key is composed of the following elements:

hash(video_path) + hash(fps) + hash(max_frames) + hash(prompt)

A cache HIT occurs only if the video, FPS, max_frames, and the prompt are all identical. Changing any one of these parameters will result in a cache MISS.

Prefix caching reuses KV cache for repeated prompts by storing them in fixed-size blocks. When multiple users provide the same system prompt, they share the same underlying blocks, incrementing a reference count (ref_count++).

Cache Key Strategies:

• LLM: hash(prompt)
• Images: hash(image_content) + hash(prompt)
• Videos: hash(video_path) + hash(fps) + hash(max_frames) + hash(prompt)

vLLM-MLX provides two engine types depending on your performance and concurrency requirements:

• Simple Engine: A direct wrapper around mlx-lm or mlx-vlm. It is optimized for maximum throughput for a single user and has zero batching overhead.
• Batched Engine: Uses AsyncEngineCore with continuous batching. This is designed for multiple concurrent requests and utilizes a scheduler with a priority queue to manage them.

vLLM-MLX provides several advanced capabilities for local LLM serving on Apple Silicon:

• Multimodal Support: Unified platform for Text, Image, Video, and Audio.
• OpenAI API Compatibility: Acts as a drop-in replacement for the OpenAI client.
• Embeddings: Provides an OpenAI-compatible /v1/embeddings endpoint.
• MCP Tool Calling: Integration with external tools via the Model Context Protocol.
• Performance Optimizations: Includes Paged KV Cache for memory-efficient prefix sharing and Continuous Batching for high throughput with concurrent users.
• Native TTS: Supports multiple languages including Spanish, French, Chinese, and Japanese.

Use --moe-top-k when:

• You are running a Qwen3 MoE (or compatible models like Qwen3.5 MoE, Gemma-MoE) and single-user decode throughput is your primary bottleneck.
• You can tolerate a small quality drop in exchange for lower latency.
• You are deploying on M-series Apple Silicon where expert gather dominates per-step decode time.

Skip --moe-top-k when:

• You are serving dense models (the flag is a no-op).
• You require maximum leaderboard-level accuracy.
• You are running long chain-of-thought or "thinking mode" generations, where the quality loss may be more severe than standard 0-shot benchmarks suggest.
• You are already using aggressive quantization (e.g., 3-bit) combined with low top_k, as this can cause the model's output quality to break entirely.