semble

Run benchmarks measuring recall vs. token budget. Requires the benchmark extra and rg on your $PATH.

Tasks:

• recall: Runs the benchmark and generates data.
• plot: Regenerates plots from a saved recall payload.

# Run recall benchmark
uv run python -m benchmarks.token_efficiency recall
uv run python -m benchmarks.token_efficiency recall --repo fastapi

# Regenerate plots
uv run python -m benchmarks.token_efficiency plot

For scripted or sandboxed environments, you can skip interactive prompts using the --agent and --type flags.

• --agent: Accepts one or more agent IDs (e.g., claude, codex, pi).
• --type: Accepts mcp, instructions, subagent, or all (default: all).
• --yes: Skips confirmation prompts (requires --agent for non-interactive runs).

semble install --agent claude --type mcp subagent --yes

Generate speed vs. NDCG plots. This writes speed_vs_ndcg_cold.png and speed_vs_ndcg_warm.png to the assets/images/ directory.

uv run python -m benchmarks.plot

Run benchmarks for probe. This requires the probe binary to be available on your $PATH (e.g., via npm install -g @buger/probe).

uv run python -m benchmarks.baselines.probe
uv run python -m benchmarks.baselines.probe --repo fastapi --repo axios

Run benchmarks for ColGREP. This requires the colgrep binary on your $PATH.

uv run python -m benchmarks.baselines.colgrep --repo fastapi --repo axios

Run benchmarks focused on speed (index time and query latency). Results are written to benchmarks/results/speed-<sha12>.json.

uv run python -m benchmarks.speed_benchmark

The recommended way to install Semble is using the interactive semble install command. This detects your installed agents and configures three types of integrations:

1. MCP server: Exposes Semble as a native tool for your agent.
2. AGENTS.md/CLAUDE.md: Adds a usage guide to the agent's configuration file.
3. Sub-agent: Installs a dedicated semble-search sub-agent for supported harnesses.

Prerequisite for Pi: You must install the MCP extension before running the Semble installer.

pi install npm:pi-mcp-extension

uv tool install semble
semble install

For scripted or sandboxed environments, use semble install with flags to skip interactive prompts.

• --agent <id>: One or more agent IDs (e.g., claude, codex, pi). Use lowercase.
• --type <type>: One or more integration types: mcp, instructions, subagent, or all (default: all). Requires --agent.
• -y or --yes: Skips confirmation prompts. Requires --agent for a fully non-interactive run.

semble install --agent claude pi --type mcp subagent --yes

Run benchmarks for grepai. This requires grepai on your $PATH and an active Ollama daemon with the nomic-embed-text model pulled.

Prerequisites:

ollama pull nomic-embed-text

Options:

• --timeout <seconds>: Sets the timeout for indexing (default 120). Use higher values for large repos.
• --output <path>: Enables resume mode; completed repos are skipped on restart.

uv run python -m benchmarks.baselines.grepai --repo fastapi --repo axios

# Example with custom timeout and output for resume mode
uv run python -m benchmarks.baselines.grepai --timeout 1800 --output results.json

Before running benchmarks, you must clone or update the pinned repositories. These are stored in ~/.cache/semble-bench and are defined in repos.json.

uv run python -m benchmarks.sync_repos          # clone / update
uv run python -m benchmarks.sync_repos --check  # verify only

Run benchmarks for CodeRankEmbed. This requires the benchmark extra to be installed via uv sync --extra benchmark.

uv run python -m benchmarks.baselines.coderankembed --mode semantic