sparkrun Documentation

The sparkrun proxy is a unified, OpenAI-compatible gateway that discovers running sparkrun inference endpoints and exposes them through a single API powered by LiteLLM. It acts as a front-end for one or more inference workloads, providing:

• Live endpoint discovery: Automatically finds running workloads using the same mechanism as sparkrun cluster status.
• Auto-discovery: A background process that periodically re-scans and syncs models to handle drift.
• Health checking: Validates endpoints via GET /v1/models.
• Deduplication: Ensures endpoints reachable on multiple interfaces (e.g., management vs. ConnectX-7) are not duplicated.
• Model aliases: Allows clients to address models by friendly names.
• Load/unload management: Syncs the proxy with the current set of running models.

Note on configuration updates: Because the proxy uses a generated config file as the single source of truth, applying changes (like adding a model or alias) requires rewriting the config and restarting the process. The proxy is optimized to skip the restart if the desired state already matches the disk.

When requesting a recipe, sparkrun follows a specific search order to locate the configuration file:

1. @registry/recipe-name: Scoped lookup within a specific registry.
2. URL: Fetched via HTTP/HTTPS and cached.
3. File path: An exact path or a path with a .yaml/.yml extension.
4. CWD scan: Scans the Current Working Directory for valid .yaml/.yml files (files must contain model, container, and a resolvable runtime).
5. Registry search: A flat and recursive lookup performed independently for each enabled registry.

Disambiguation

• Across Registries: Use the @registry/name syntax to specify which registry to use.
• Within a Registry: Use @registry/subdir/name to specify a path under the registry's recipe directory (the scope is split on the first /).
• Ambiguity Errors: If a name matches multiple recipes within a single registry (e.g., a/foo.yaml and b/foo.yaml), sparkrun will error and provide the path-qualified names. If your terminal is a TTY, the CLI will instead provide a numbered prompt to select the correct one.

Example error message: Recipe 'foo' is ambiguous — 2 matches in registry 'official' (@official/a/foo, @official/b/foo). Use the full name to specify.

Understanding the following abstractions is key to managing workloads via OpenClaw:

• Recipes: YAML files describing an inference workload (model, runtime, container, defaults). Recipe names can use @registry/name syntax for explicit registry selection.
• Runtimes: Inference engines such as vLLM, SGLang, llama.cpp, or TensorRT-LLM.
• Clusters: Named groups of DGX Spark hosts.
• Registries: Git-based collections of recipes and benchmark profiles.
• Benchmark profiles: Standardized benchmark configurations defined within registries.
• Proxy: A unified OpenAI-compatible gateway that sits in front of multiple inference endpoints.
• Tensor Parallelism (TP): Since each DGX Spark has 1 GPU, the --tp N flag maps directly to N hosts.
• Detached Containers: sparkrun launches containers in a detached state. Using Ctrl+C will detach from logs but will not kill the running job.

Recipes are YAML files that define an inference workload for NVIDIA DGX Spark.

Required Fields

• model: HuggingFace model ID.
• runtime: One of vllm, sglang, or llama-cpp.
• container: The Docker image and tag (e.g., registry/image:tag).

Key Configuration Sections

• defaults: Defines parameters like port, host, tensor_parallel, pipeline_parallel, gpu_memory_utilization, and max_model_len. These are used for auto-generating commands and can be overridden via CLI.
• command: An optional explicit command template. It supports {placeholder} syntax (e.g., {model}, {tensor_parallel}) which is substituted from defaults or CLI overrides.
• env: Environment variables passed to the container. Supports shell variable expansion (e.g., ${HF_TOKEN}).
• post_exec: Commands to run inside the head container after the server is healthy.
• post_commands: Commands to run on the control machine after the server is healthy.

Important Notes

• GGUF + SGLang: When using GGUF models with the sglang runtime, you must set tokenizer_path in the defaults section, pointing to the base non-GGUF model.
• Parallelism: pipeline_parallel enables pipeline parallelism. The total nodes required is calculated as TP * PP.

model: org/model-name
runtime: vllm
container: registry/image:tag
min_nodes: 1
max_nodes: 4
model_revision: abc123

metadata:
  description: Human-readable description
  maintainer: name <email>
  model_params: 7B
  model_dtype: fp16
  category: general

defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 2
  pipeline_parallel: 1
  gpu_memory_utilization: 0.9
  max_model_len: 32768
  served_model_name: my-model
  tokenizer_path: org/base-model

command: |
  python3 -m vllm.entrypoints.openai.api_server \
      --model {model} \
      --tensor-parallel-size {tensor_parallel} \
      --port {port}

env:
  NCCL_DEBUG: INFO

post_exec:
  - "echo 'Model loaded'"
post_commands:
  - "curl http://{head_ip}:{port}/v1/models"
stop_after_post: false

To effectively manage workloads with sparkrun, understand these key abstractions:

• Recipes: YAML files that describe an inference workload, including the model, runtime, container, and default settings. Recipe names can use @registry/name syntax for explicit registry selection.
• Runtimes: The inference engines supported, such as vLLM, SGLang, llama.cpp, and TensorRT-LLM.
• Clusters: Named groups of NVIDIA DGX Spark hosts.
• Registries: Git-based collections containing recipes and benchmark profiles.
• Benchmark profiles: Standardized configurations for running benchmarks, sourced from registries.
• Proxy: A unified, OpenAI-compatible gateway (based on LiteLLM) that sits in front of multiple inference endpoints.
• Tensor Parallelism (TP): Since each DGX Spark has 1 GPU, the --tp N flag maps directly to the number of hosts (e.g., --tp 2 uses 2 hosts).
• Detached Containers: sparkrun launches containers in a detached state. Using Ctrl+C will detach you from the logs but will not kill the running job.

OpenClaw automatically activates specific skills when your natural language request matches the task context. These skills allow OpenClaw to interact with the sparkrun CLI and the sparkrun_exec tool.

Usage Examples

You can interact with your cluster using natural language commands such as:

• "Run the Qwen3 1.7B model on my cluster"
• "What inference jobs are running?"
• "Stop all inference jobs on my cluster"
• "Show me available recipes for llama models"
• "Benchmark the sglang recipe on a single node"
• "Set up sparkrun on my DGX Spark cluster"
• "Configure CX7 networking on my cluster"
• "Create a recipe for Mistral 7B on vLLM"
• "Monitor my cluster's GPU usage"
• "Start the inference proxy and load a model"