Lucebox Documentation

When using Spark, keep the following constraints and requirements in mind:

• Hardware Target: Validated for a Single 24 GB GPU (specifically Laguna-XS.2 Q4_K_M). Other MoE models require re-calibration.
• Traffic Calibration: You must calibrate on the specific traffic you intend to serve. A profile optimized for one distribution (e.g., conversation) will under-perform on another (e.g., pure-code).
• Locality Requirement: Spark relies on workload locality. The cache is effective because agent traffic typically has a small, stable working set. Workloads without locality will suffer from PCIe-bound performance.
• Performance Ceiling: Achieving full all-GPU speed requires model changes. A fitted pre-gate caps at approximately 53% recall@8 due to the pre/post-attention information gap.

Lucebox implements two tiers of prefix caching to reduce latency and compute:

Tier 1: Inline Prefix Cache

Caches KV state at turn boundaries within a conversation. It uses ChatMarkers to detect end-of-message and start-of-next-role sequences. This is particularly effective for tool-using chat templates where tool definitions are part of the system prefix; the first request pays the prefill cost, and subsequent requests restore it.

Tier 2: Exact Prefill Cache

Caches the entire post-compression KV state in RAM, keyed on the SHA-1 hash of the raw (pre-compression) prompt. This is the fastest path as it skips both PFlash compression and the prefill step entirely. It uses a separate slot pool starting at the index defined by --prefix-cache-slots.

The DFlash server includes several advanced inference features:

• Multi-turn conversation: Full message history support.
• Tool/function calling: Supports both XML and JSON tool parsing.
• Stop sequences: Supports OpenAI stop and Anthropic stop_sequences.
• Thinking/reasoning: Controlled via OpenAI reasoning.effort or Anthropic thinking.type.
• Prefix cache: Automatic KV cache reuse (supports both memory and persistent disk caching).
• PFlash: Speculative prefill to compress long prompts.
• Client disconnect detection: Aborts generation if the client disconnects.
• CORS: Enabled by default.

The Jinja engine is a C++ implementation consisting of several decoupled components:

• jinja::lexer: A predictive parser that converts Jinja source code into a list of tokens. It processes source as-is to allow for source tracing on errors.
• jinja::parser: Consumes tokens to compile them into a jinja::program (AST).
• jinja::runtime: Executes the compiled jinja::program using a provided context by recursively calling execute(ctx) on statements and expressions.
• jinja::value: Defines primitive types (int, float, bool, string, array, object, none, undefined) and built-in functions. It uses shared_ptr to manage values and avoid C++ operator overloading for clarity.

The bsa_stubs directory provides header shims that allow mit-han-lab/Block-Sparse-Attention (BSA) to compile without a dependency on PyTorch's libtorch. This is used in the dflash daemon to satisfy specific header references required by BSA without linking the entire PyTorch library.

Key stubs provided:

• c10/cuda/CUDAException.h: Provides C10_CUDA_CHECK and C10_CUDA_KERNEL_LAUNCH_CHECK macros (forwarded to cudaPeekAtLastError).
• ATen/cuda/CUDAGeneratorImpl.h: Provides the at::PhiloxCudaState POD struct.
• ATen/cuda/CUDAGraphsUtils.cuh: Provides a no-op at::cuda::philox::unpack function.

Note: BSA is built with FLASHATTENTION_DISABLE_DROPOUT, so the philox/generator stubs are only present to satisfy declarations and are never actually exercised.

Lucebox Hub uses a specific, vendored subset of ggml located in server/deps/llama.cpp/. This is not a full llama.cpp installation but a specialized ggml-only subset designed for the Hub's execution requirements.

Key characteristics of this snapshot:

• Source Base: Derived from the luce-dflash branch of https://github.com/Luce-Org/lucebox-ggml at commit 6fbe72d67069136bbd370be703e1d4f441b5e942.
• Included Patches: Contains specific merged PRs (#35, #37) and upstream backport llama.cpp #22298.
• Vendored Paths: The following directories/files are included: LICENSE, common/jinja, common/log.h, common/unicode.*, ggml, and gguf-py.

Model cards are sidecar JSON files used by dflash_server to provide per-model defaults. These defaults are transcribed from upstream model cards (such as HuggingFace READMEs and generation_config.json).

At startup, dflash_server reads these files to automatically set sensible values for:

• --default-max-tokens
• --think-max-tokens
• Sampler settings
• reasoning.effort tier values

Note: CLI flags will always override the values provided in these model cards.

If you prefer to manage client binaries manually or want to avoid automatic installation during launcher execution, you can preinstall specific clients using the client_test_runner.py script.

python3 harness/client_test_runner.py install --clients codex,hermes,openwebui

By default, Lucebox trims Claude-specific prompt boilerplate to improve local-model reliability. To bypass this and test the raw Anthropic Messages/User prompts, set the following environment variables:

• DFLASH_ANTHROPIC_RAW_SYSTEM=1
• DFLASH_ANTHROPIC_RAW_USER=1

DFLASH_ANTHROPIC_RAW_SYSTEM=1 DFLASH_ANTHROPIC_RAW_USER=1 \
  harness/clients/run_claude_code.sh

The DFlash server implements the Responses API (/v1/responses) required by OpenAI Codex. Follow these steps to set up the integration:

1. Start the DFlash server

Run the dflash_server binary providing the target model and the draft model. Use the --ddtree flag to enable the draft tree.

2. Configure Codex

Edit ~/.codex/config.toml to point to your local DFlash instance. No env_key is required for local servers.

3. Run Codex

Use the codex command specifying the dflash provider.

# 1. Start the DFlash server
./build/dflash_server models/Qwen3.5-27B-Q4_K_M.gguf \
  --draft models/Qwen3.5-3B-f16.safetensors \
  --ddtree --ddtree-budget 22 --port 8080

# 2. Configure Codex (~/.codex/config.toml)
# [toml content below]

# 3. Run Codex
codex --provider dflash "Explain this codebase"

For optimal performance during the compress phase, use a small, fast drafter.

• Recommended: Qwen3-0.6B in BF16 safetensors (with ~5 attention layers).
• Strategy: Use the Qwen3-0.6B BF16 drafter for the compress (PFlash) side, and reuse the larger DFlash drafter for the decode (DFlash) side after unparking. This prevents loading two large drafters simultaneously on a 24 GB GPU.