BenchLocal Documentation

BenchLocal is a local-first desktop application designed for running, comparing, and managing installable LLM Bench Packs against both local and remote models. It provides a shared desktop runtime that manages:

• Provider configuration
• Model registry
• Bench Pack installation and updates
• Per-tab sampling overrides
• Run execution and result history
• Verifier lifecycle management
• Persisted desktop UI state

Each Bench Pack is self-contained and owns its own scenario definitions, prompts, scoring logic, verifier contracts, and traces.

@benchlocal/core is the low-level foundation of the BenchLocal ecosystem. It defines the shared data structures, protocols, and storage shapes that the BenchLocal desktop app, Bench Pack host runtime, and Bench Pack SDK all use to communicate.

When to use it:

• Bench Pack Authors: You should generally use @benchlocal/sdk instead of @benchlocal/core for standard development. Only use @benchlocal/core if you need direct access to the core protocol or configuration types.
• App/Host Developers: Use it to implement the BenchLocal desktop app or Bench Pack host runtime.
• Tooling Developers: Use it if you are building ecosystem tools that need to read or validate BenchLocal configuration or workspace state.

When you run BenchLocal, the system follows a specific lifecycle to prepare the environment for benchmarking:

1. Initialization: BenchLocal loads configuration from ~/.benchlocal/config.toml and current state from ~/.benchlocal/state.json.
2. Inspection: It scans the ~/.benchlocal/benchpacks directory for installed Bench Packs.
3. UI Rendering: The renderer displays available workspaces, tabs, providers, models, and Bench Packs.
4. Run Execution:
   • Resolves the active Bench Pack and selected models for the current tab.
   • Resolves provider secrets.
   • Starts any required verifiers.
   • Constructs a HostContext.
   • Loads the Bench Pack's runtime entrypoint.
5. Execution & Persistence: The Bench Pack executes scenarios and emits progress events. BenchLocal then persists logs, summaries, and history to ~/.benchlocal/runs.

The @benchlocal/web-sdk is designed for Interactive Web Bench Packs that run in a browser surface inside the BenchLocal desktop app.

Mental Model:

• The Web Page (Your App): Owns the visual experience, custom UI, progress bars, and the benchmark workflow.
• BenchLocal (The Desktop App): Owns sensitive execution, including provider credentials, model configuration, and the actual inference engine.
• Communication: The SDK communicates via a postMessage bridge. It is not a general HTTP proxy and does not expose API keys to the web page.

Use this SDK when you need richer presentation than a standard table (e.g., form-filling, agent demos, or visual tool-use tests). For standard table-based benchmarks, use @benchlocal/sdk instead.

BenchLocal is a desktop host designed to install, configure, and run LLM Bench Packs. The system is divided into two distinct layers with clear ownership boundaries:

BenchLocal (The Host)

Responsible for the shared runtime and desktop environment, including:

• Desktop UI and Electron processes.
• Provider and model registries.
• Bench Pack installation, updates, and state management.
• Verifier lifecycle management (startup, health checks, and Docker lifecycle).
• Run orchestration, persistence, and history.
• Shared user configuration and local storage.

Bench Packs (The Content)

Responsible for benchmark-specific logic, including:

• Scenarios, prompts, and tool logic.
• Scoring logic and results (traces and summaries).
• Verifier implementation and request/response contracts.
• Recommended default sampling values and scenario metadata.

Bench Packs that require external validation declare their requirements in benchlocal.pack.json. The system uses a shared responsibility model for verifiers:

BenchLocal Responsibilities

• Mode Selection: Supports docker, cloud, or custom_url.
• Lifecycle: Manages local Docker lifecycles and health checks.
• Networking: Handles dynamic host port assignment.
• UI: Displays verifier status to the user.

Bench Pack Responsibilities

• Implementation: Provides the actual verifier code.
• Contract: Defines the request and response schema.
• Endpoint Usage: Uses the resolved verifier endpoint.

Crucial Networking Detail: The Bench Pack declares the verifier's internal listenPort, but BenchLocal automatically assigns the actual host port to avoid conflicts.

For non-verifier packs that need to interact with an external agent runtime (e.g., one expecting an OpenAI-compatible URL), do not wire provider secrets directly into the pack runtime. Instead, use the host-provided helpers to maintain ownership of credentials by BenchLocal.

Recommended Pattern: Use createHostHelpers(context).getRequiredInferenceEndpoint(modelId).

If your runtime lives inside a Docker verifier, forward dockerBaseUrl ?? baseUrl to the verifier.

BenchLocal uses two distinct files for configuration and state, stored in ~/.benchlocal/:

• config.toml: Contains durable configuration such as providers, models, installed Bench Packs, verifier preferences, and theme selection.
• state.json: Contains transient UI state such as workspaces, tabs, selected models per tab, sampling overrides, and execution modes.

The directory structure is as follows:

~/.benchlocal/
  config.toml
  state.json
  benchpacks/
  runs/
  logs/
  cache/
  themes/

Interactive Web Bench Packs are a second presentation model in BenchLocal. Unlike standard table packs that render results in a table, web packs use a hosted web application to provide a richer, interactive benchmark experience.

Key characteristics:

• Local Authority: BenchLocal remains the local authority for credentials, providers, models, inference, history, and artifacts.
• Security: The hosted web app is sandboxed and does not have access to provider credentials or the local filesystem.
• Execution: Inference is performed through the local BenchLocal provider/model configuration, not by the web app itself.
• Compatibility: web packs are an evolution of the protocol and coexist with existing table packs.