semtools

The ArXiv benchmark dataset consists of 1000 papers organized by author, category, date, and full text. You can obtain the dataset in two ways:

1. Download the full dataset directly: Use the provided OneDrive link.
2. Run the download script: Use the download_arxiv_files.py script located in the benchmark directory to download and organize the files locally.

The resulting directory structure follows this pattern:

├── by_author/
│   ├── Aadhrik_Kulia
│   │   ├── 2507.22047v1_fulltext.txt
│   │   └── ...
│   └── ...
├── by_category/
│   ├── cs.AI
│   │   ├── 2505.20278v1_fulltext.txt
│   │   └── ...
│   └── ...
├── by_date/
│   ├── 2025-05
│   │   ├── 2505.20277v2_fulltext.txt
│   │   └── ...
│   └── ...
├── full_text/
│   ├── 2505.20277v2.txt
│   └── ...

To give an agent full potential with semtools, it is recommended to provide direct terminal access rather than wrapping individual commands. You can use fastmcp to create a server that executes bash commands.

WARNING: This approach is unsafe for production. It is highly recommended to run this inside a Docker container with mounted volumes and restricted user permissions to prevent unrestricted access to your host machine.

# Install fastmcp
pip install fastmcp

# Create server.py
import subprocess
from fastmcp import FastMCP

mcp = FastMCP("My MCP Server")

@mcp.tool
def execute_bash(command: str) -> str:
    """Useful for executing bash commands on your machine.""
    return subprocess.run(command, shell=True, capture_output=True, text=True, timeout=60).stdout

# Launch the server
fastmcp server.py:mcp --transport streamable-http

You can augment the capabilities of coding agents (such as Claude-Code, Cursor, or Gemini CLI) by providing them access to semtools. This allows agents to:

• Parse unsupported files: Convert files like PDFs or Word docs into searchable formats.
• Perform semantic search: Find relevant information across large datasets using semantic similarity.
• Create pipelines: Chain semtools parse and semtools search with other CLI commands to automate complex research tasks.

Setup Workflow

1. Install the CLI via Cargo.
2. Configure API access: Set the LLAMA_CLOUD_API_KEY environment variable.
3. Seed Agent Knowledge: Add instructions and usage examples for semtools to your agent's configuration file (e.g., CLAUDE.md or AGENTS.md) so the agent knows how to invoke the tool.

To execute the benchmark, follow these steps:

1. Prepare the environment: Place the CLAUDE.md file you wish to test (either a plain version or a version with search capabilities) into the arxiv_dataset_1000_papers directory.
2. Execute questions: Use claude-code to prompt the questions listed in questions.txt.
3. Workflow: Each question should be asked in a fresh chat session.
4. Record results: Copy and paste the responses from claude-code into the answers directory.

Note: The benchmark was originally validated using cloud-code v1.0.90 and SemTools v1.2.0.

When searching, you can control the amount of context returned or use a distance threshold instead of a fixed top-k count.

• --n-lines <N>: Controls context around matches. Tip: The default (3) is often too small; consider using 30-50 for better results.
• --max-distance <FLOAT>: Returns all results with a distance below this threshold. Useful when you don't know the required top-k value.

Workspaces accelerate search over large collections by caching embeddings. If you plan to run repeated searches over the same files, you should use a workspace to avoid re-embedding documents from scratch every time.

Workflow:

1. Create/select a workspace: semtools workspace use <name>
2. Activate it: export SEMTOOLS_WORKSPACE=<name>
3. Run search/parse commands. The embeddings will be cached in ~/.semtools/workspaces/.

The search command is a CLI tool designed for fast semantic keyword search using static embeddings. It functions similarly to grep but provides fuzzy/semantic matching capabilities rather than just exact pattern matching. It can operate on a list of files/directories or via stdin.

When to use search vs grep:

• Use grep when looking for exact matches, known patterns, or when you need to filter a large volume of results produced by search.
• Use search when looking for fuzzy matches, patterns you don't know exactly, or when you want to semantically filter results.
• Hybrid approach: For large datasets, use grep with a generic pattern first to narrow down files, then pipe to search for semantic filtering.

search "machine learning" *.txt --n-lines 6 --max-distance 0.4 --ignore-case

To use semtools commands like parse and search within an agentic workflow, install the binary using cargo install.

cargo install semtools

You can install semtools via npm or cargo.

Note: Installing via npm builds the Rust binaries locally, which requires Rust and Cargo to be available in your environment. If you don't have them, install via rustup at https://www.rust-lang.org/tools/install.

Via npm

npm i -g @llamaindex/semtools

Via cargo

To install the entire crate:

cargo install semtools

To install only select features (e.g., just the parse feature):

cargo install semtools --no-default-features --features=parse

The Store tracks document changes by comparing the current filesystem metadata against stored DocMeta. A document is considered Changed if any of the following occur:

1. Size Mismatch: The size_bytes on disk differs from the stored value.
2. Timestamp Mismatch: The mtime (modification time) on disk differs from the stored value.
3. Version Mismatch: The _version field in the stored metadata does not match the CURRENT_EMBEDDING_VERSION used during the analysis.

If the file is not found in the store, it is marked as New. If all metadata fields match, it is marked as Unchanged.

SemTools uses a unified configuration file at ~/.semtools_config.json. You can also specify a custom path using the -c or --config flag.

Configuration Priority:

1. CLI arguments
2. Config file
3. Environment variables
4. Built-in defaults

Example Configuration:

{
  "parse": {
    "api_key": "your_llama_cloud_api_key_here",
    "num_ongoing_requests": 10,
    "base_url": "https://api.cloud.llamaindex.ai",
    "parse_kwargs": {
      "tier": "agentic",
      "version": "latest",
      "disable_cache": false
    }
  },
  "ask": {
    "api_key": "your_openai_api_key_here",
    "model": "gpt-4o-mini",
    "max_iterations": 20,
    "api_mode": "responses"
  }
}

Environment Variables:

• LLAMA_CLOUD_API_KEY: For the parse tool.
• OPENAI_API_KEY: For the ask tool.