spark-vllm-docker

The EAGLE Fine Prefix Fix is a runtime-only modification for vLLM that restores fine-grained hybrid prefix-cache hits when using MTP (Multi-Token Prediction) or EAGLE on full-attention + Mamba hybrid models (such as Qwen 3.5/3.6).

Standard vLLM behavior (after PR #46384) registers only the final prompt-tail hash at the prefix_match_unit granularity. This causes follow-up turns to miss cache entries if a context-load probe (like a single . token) occupies the last hash unit. This mod retains the predecessor FullAttention hash and the Mamba replay boundary, allowing EAGLE to rewind to a valid match even when the immediate tail has diverged.

Recipes allow you to launch models with preconfigured settings using a single command.

Common run-recipe.sh commands:

• List available recipes: ./run-recipe.sh --list
• Run in solo mode (single node): ./run-recipe.sh <recipe-name> --solo
• Full setup (build container + download model + run): ./run-recipe.sh <recipe-name> --solo --setup
• Run with overrides: ./run-recipe.sh <recipe-name> --solo --port 9000 --gpu-mem 0.8

The --earlyoom flag allows you to run earlyoom as the container's PID 1. This monitors memory pressure and can prevent system instability by killing processes before the host runs out of memory. This is useful when launching vLLM via docker exec.

Default Policy

earlyoom -M 524288,102400 -s 100 -r 60

• -M 524288,102400: SIGTERM at 512 MiB available, SIGKILL at 100 MiB.
• -s 100: Acts on host RAM pressure without waiting for swap to fill.
• -r 60: Reports every 60 seconds.

Customizing the Policy

Use --earlyoom-args to pass custom arguments to earlyoom. Note that --earlyoom is incompatible with --keep-entrypoint because the launcher must clear the image entrypoint to allow earlyoom to run as PID 1.

./launch-cluster.sh --earlyoom \
  --earlyoom-args "-M 1048576,262144 -s 70,50 -r 30" \
  exec vllm serve ...

This mod is a vLLM integration workaround rather than a native selective-loading implementation within InstantTensor.

Key technical details:

• Metadata Scanning: Lazy safetensors still performs a scan of the checkpoint metadata.
• Memory Management: Weights rejected by the draft model remain as CPU memory-mapped views; they are not staged or cloned onto the GPU.
• Lifecycle: This mod should be removed once vLLM provides a native per-draft load-format option or when InstantTensor and vLLM can natively coordinate selective tensor streaming.

The mod works by patching the Inkling-specific attention operator and vendoring the Python/CuTe DSL source from SecondNatureComputing/flash-attn-4-sm120.

Key technical details:

• Target: Only vllm.models.inkling.nvidia.ops.fa4_rel_attention is patched.
• Architecture: Only compute capability major 12 triggers the alternate dispatch.
• Constraint: num_splits is forced to 1.
• Compatibility: The mod includes necessary CUTLASS DSL 4.6 API migrations (cute.core.ThrMma $\rightarrow$ cute.ThrMma and cute.make_fragment $\rightarrow$ cute.make_rmem_tensor) required for CUDA 13 vLLM images.
• Safety: The run.sh script validates the environment (Inkling source shape, FA4 dependencies, CUTLASS DSL APIs, CUDA 12.8+, and SM12x) before patching. If requirements aren't met, it fails without changing the model dispatch. Reapplying the mod is safe.

launch-cluster.sh automatically parses parallelism arguments (-tp / --tensor-parallel-size, -pp / --pipeline-parallel-size, and -dp / --data-parallel-size) from the exec command or launch script. It then adjusts the active node count:

• If fewer nodes are needed than configured: Only the required nodes get containers started; excess nodes remain idle.
• If more nodes are needed than available: An error is raised before starting.

No extra flags are required for this check to occur.

Solo Mode (Single Node)

Use --solo to run a model on a single node without Ray. If no nodes are configured in .env, the runner defaults to solo mode.

Cluster Mode (Multiple Nodes)

Use -n to specify nodes directly (the first IP is the head node) or use --discover to use nodes from .env.

When using cluster mode with --setup, the container is built locally and copied to all worker nodes, and the model is downloaded locally and copied to all worker nodes.

Cluster-Only Recipes

Some recipes are marked with cluster_only: true because the model is too large for a single node. Attempting to run these with --solo will result in an error.

Launch scripts are standard bash scripts. When executed, the script is copied into the container at /workspace/exec-script.sh.

It is recommended to include metadata comments for documentation:

• # PROFILE: A human-readable name.
• # DESCRIPTION: A description of what the script does.

You can also set environment variables within the script before running your primary command (e.g., vllm serve).

#!/bin/bash
# PROFILE: MiniMax-M2-AWQ Example
# DESCRIPTION: vLLM serving MiniMax-M2-AWQ with Ray distributed backend

# Optional: Set environment variables
export MY_VAR="value"

# Run your command
vllm serve org/model-name \
    --port 8000 \
    --host 0.0.0.0 \
    --gpu-memory-utilization 0.8

The mod benefits workloads where a 'Producer' request (A) has a unique tail that fits within the coverage window defined by the prefix_match_unit (H). A subsequent 'Consumer' request (B) can then reuse the shared prefix.

Qualifying Workloads:

• Large System/Developer Prompts: Reusing large system contexts, tool schemas, or catalogs where the user turn is relatively short.
• Conversation Forks/Edited Turns: Branching from a common history (e.g., editing a user turn or retrying an instruction) where the first branch's continuation is short.
• Agentic/Tool Workflows: Reusing agent policies and working states where the tool result or next-action suffix is within the window.
• RAG & Few-Shot: Reusing fixed instruction sets or document corpora across different queries.
• Context Probes: Using tiny tokens (like . in llama-benchy) to warm the cache.

DGX Spark has a unique ConnectX setup where the hardware limitation of the SOC (max x4 PCIe lanes per device) means a single physical port shares a pair of PCIe 5.0 x4 connections. This results in two 'twin' Ethernet and RoCE interfaces for a single cable connection.

To achieve full bandwidth in NCCL RDMA mode for vLLM, you must utilize both RoCE twins. This is done by setting the NCCL_IB_HCA environment variable to include both interfaces.

Example twin pairs for a single cable:

• Ethernet: enp1s0f1np1 and enP2p1s0f1np1
• RoCE/IB: rocep1s0f1 and roceP2p1s0f1

Note: ./launch-cluster.sh handles this autodiscovery and configuration automatically if your Ethernet interface is set up correctly.

export NCCL_IB_HCA=rocep1s0f1,roceP2p1s0f1