oMLX LLM Inference Server

oMLX is a high-performance serving engine for Apple Silicon that supports a wide range of models including Text LLMs, Vision-Language Models (VLM), OCR models, Embedding models, and Rerankers.

Key features include:

• Admin Dashboard: A web UI at /admin for real-time monitoring, model management, benchmarking, and configuration. It supports multiple languages and operates fully offline.
• Hierarchical KV Cache: A block-based cache system (inspired by vLLM) that uses a Hot Cache (RAM) for frequent blocks and a Cold Cache (SSD) for overflow, allowing for prefix sharing and persistence across server restarts.
• Continuous Batching: Uses mlx-lm's BatchGenerator to handle concurrent requests efficiently.
• Multi-Model Serving: Supports loading multiple model types simultaneously with features like LRU Unloading (to free memory), Model Pinning (to keep models loaded), and Model-specific TTL (idle timeouts).
• API Compatibility: Provides drop-in replacements for OpenAI and Anthropic APIs, including support for streaming usage statistics, adaptive thinking, and vision inputs.

oMLX is a high-performance inference server for Apple Silicon that supports text LLMs, vision-language models (VLM), OCR models, embeddings, and rerankers.

Key features include:

• Admin Dashboard: A web UI at /admin for real-time monitoring, model management, chat, and benchmarking. It supports multiple languages and operates fully offline via vendored CDN dependencies.
• Tiered KV Cache: A block-based cache system (inspired by vLLM) that uses a Hot tier (RAM) for fast access and a Cold tier (SSD) in safetensors format to offload blocks when memory is full, allowing for fast restoration after restarts.
• Continuous Batching: Uses mlx-lm's BatchGenerator to handle concurrent requests.
• Multi-Model Serving: Supports loading multiple model types simultaneously with LRU eviction, manual load/unload, model pinning, and per-model TTL (Time-To-Live).
• API Compatibility: Provides a drop-in replacement for OpenAI and Anthropic APIs, supporting streaming, vision inputs, and tool calling.

oMLX is an Apple Silicon-optimized inference server supporting Text LLMs, Vision Language Models (VLM), OCR models, Embeddings, and Rerankers.

Key features include:

• Web Admin UI: Accessible at /admin for real-time monitoring, model management, benchmarking, and configuration. It supports multiple languages and works fully offline.
• Hierarchical KV Cache: Uses a block-based management system (inspired by vLLM) with a Hot Cache (RAM) for fast access and a Cold Cache (SSD) that offloads blocks in safetensors format to preserve them across server restarts.
• Continuous Batching: Processes simultaneous requests via mlx-lm's BatchGenerator.
• Multi-Model Serving: Supports loading multiple model types simultaneously with LRU eviction, manual load/unload, model pinning, and per-model TTL (Time-To-Live).
• Claude Code Optimization: Supports context scaling and SSE keep-alive to prevent timeouts during long prefills.
• Integrations: One-click setup for OpenClaw, OpenCode, Codex, Hermes Agent, Copilot, and Pi via the Admin UI.

DFlash is an experimental block diffusion speculative decoding technique (based on arXiv:2602.06036) integrated into oMLX via the DFlashEngine. Unlike traditional speculative decoding that drafts tokens one-by-one, DFlash uses block diffusion to propose a block of 16 tokens in parallel. This allows the target model to verify all 16 tokens in a single forward pass, significantly increasing throughput.

The DFlash Workflow:

1. PREFILL: The target model processes the prompt and captures hidden states.
2. DRAFT: The draft model generates a block of 16 tokens simultaneously via parallel denoising.
3. VERIFY: The target model verifies the entire block in one forward pass.
4. ACCEPT: A greedy prefix match identifies the longest matching prefix to commit.
5. REPLAY: For hybrid models (like GatedDeltaNet), the cache is rolled back using tape replay.
6. REPEAT: The process continues until max_tokens or an EOS token is reached.

1. PREFILL: target model processes entire prompt, captures hidden states
2. DRAFT:   draft model generates block of 16 tokens in parallel (block diffusion)
3. VERIFY:  target model verifies all 16 in one forward pass
4. ACCEPT:  greedy prefix match — longest matching prefix is committed
5. REPLAY:  cache rollback via tape replay for hybrid (GatedDeltaNet) models
6. REPEAT:  until max_tokens or EOS

DFlash supports both greedy and stochastic sampling via a custom implementation of sample_with_temperature(). This sampling is applied at three points: the prefill first token, the draft block, and the verify posterior.

Sampling Modes:

• temp=0 (Greedy): Identical to original greedy behavior. Every emitted token matches the target model's argmax. This is lossless and bit-for-bit reproducible.
• temp>0 (Stochastic): Both the draft and verify steps use temperature sampling. While this increases output diversity, it typically reduces the acceptance rate because the draft and target models are less likely to agree on stochastic samples, resulting in a lower speedup compared to greedy mode.

def sample_with_temperature(logits, temperature, suppress_token_mask=None):
    if temperature < 1e-5:
        return greedy_tokens_with_mask(logits, suppress_token_mask)  # greedy
    scaled = logits / temperature
    return mx.random.categorical(scaled).astype(mx.uint32)           # stochastic