Vexor

repository·main·Indexed 42 days ago

https://github.com/scarletkc/vexor

A semantic search engine for files and code that enables humans and AI coding assistants to discover files based on purpose. Vexor provides a CLI, a Python API, and a Model Context Protocol (MCP) server for integration with agents like Claude Code, Cursor, Windsurf, and Zed. It supports reusable indexes, persistent Collections for external data, and multiple embedding providers including OpenAI, Gemini, and VoyageAI.

Tokens
34.2K
Snippets
117
Records
149
Agent score
53%

What's inside Vexor

  1. Understand Vexor configuration precedence

    main

    Vexor resolves configuration using the following priority (highest to lowest):

    1. Explicit function arguments (e.g., provider, model, api_key)
    2. Per-call override via config=... parameter
    3. In-memory runtime config set by set_config_json(...)
    4. Environment overrides (e.g., VEXOR_CONFIG_JSON, OPENAI_API_KEY)
    5. The nearest project .vexor/config.json
    6. Global ~/.vexor/config.json
    7. Built-in defaults

    To ignore on-disk configuration files in a library context, pass use_config=False and provide explicit arguments or a config dict.

  2. Manage Project-Specific Configuration and Overrides

    main

    Vexor uses a hierarchical configuration system.

    1. Global Config: Stored at ~/.vexor/config.json. This is where mutating vexor config commands write data.
    2. Project Config: A config.json file located in the nearest .vexor/ directory relative to your target path. This acts as a restricted overlay on the global config.

    Key Rules

    • Restrictions: Project-level config.json files can only override settings; they cannot define new credentials or endpoints.
    • Resolution: When running search, index, config --show, or doctor, Vexor walks upward from the target path to find the nearest .vexor/ marker.
    • Inspection: Use vexor config --show to see the effective configuration and identify which file (global or project) provided each value.
  3. Configure Vexor search modes

    main

    Choose a --mode based on your search intent and the type of files you are targeting:

    • auto: Routes by file type (default).
    • name: Searches filenames only (fastest).
    • head: Searches only the first lines of files (fast).
    • brief: Uses keyword summaries (best for PRDs).
    • code: Uses code-aware chunking for .py, .js, and .ts files (best default for codebases).
    • outline: Uses Markdown headings and sections (best for documentation).
    • full: Chunks full file contents (slowest, but provides highest recall).
  4. Configure Reranking for improved search accuracy

    main

    Reranking reorders semantic results using a secondary ranker to improve precision. By default, reranking is off.

    Reranking Options

    • off: No reranking (default).
    • bm25: A lightweight lexical boost using a multilingual tokenizer. Good for low latency and CJK support.
    • hybrid: Performs full-corpus BM25 scoring and fuses it with dense ranking using reciprocal rank fusion. Use this when exact identifiers or lexical matches are critical. Scores are normalized in [0, 1].
    • flashrank: Stronger reranking using a local model. Requires installation: pip install "vexor[flashrank]". Models are cached in ~/.vexor/flashrank.
      • For multilingual/Chinese content, use: vexor config --set-flashrank-model ms-marco-MultiBERT-L-12.
    • remote: Calls a hosted HTTP endpoint. Requires configuring a URL and model via remote-rerank settings.

    Configuration Commands

    vexor config --rerank bm25
    vexor config --rerank flashrank
    vexor config --rerank hybrid
    vexor config --rerank remote
  5. How to retrieve chunk content from search results

    main

    By default, search results provide metadata and previews. To get the actual source text of a match, you must use either the --content flag or --format json.

    Text is read from the file at search time using the indexed line range, ensuring it reflects the current state of the disk.

    Content Availability and Limits

    Content may be unavailable (content_unavailable) for the following reasons:

    • no_line_range: The mode used (e.g., name, head, brief) does not record line ranges, or the file format (PDF, docx, etc.) lacks stable lines.
    • stale_line_range: The file has changed since it was indexed. Fix: Run vexor index to refresh.
    • budget_exhausted: The character budget was used by higher-ranked results.
    • unreadable: The file was deleted, is unreadable, or failed to decode.

    Constraints:

    • Content is capped at 2000 characters per result.
    • Total response content is capped at 8000 characters (spent on highest-ranked matches first).
    • If a result is cut short, content_truncated will be true and content_end_line will report the last line returned.
  6. What are Collections in Vexor?

    main

    A Collection is a named, persistent set of caller-owned text records. Unlike file indexes (which use vexor index to discover files on a filesystem), Collections are intended for use when your source of truth is an external system like a database or message store that already has stable record IDs.

    Collections are accessed via the Python API and store data in collections.db, separate from file indexes.

    Key Constraints:

    • The provider, model, and vector dimension are fixed upon the first successful non-empty write.
    • Subsequent writes or searches that attempt to use a different configuration (different model or dimension) will be rejected with an error. You must recreate the collection to change these settings.
  7. Understand Vexor storage and configuration locations

    main

    Vexor uses specific locations for global and project-specific data:

    • Global Cache/Config: Located in ~/.vexor.
    • Project-specific Index: If a project contains a .vexor/ directory, the index is kept there. This directory may also contain a config.json overlay (see docs/configuration.md).
    • Index Database: index.db contains index metadata, paths, BM25 postings, and query caches.
    • Dense Vectors: Stored as vectors/*.npy sidecars next to the database to allow NumPy mmap loading.
    • Text Embedding Strategy: The text embedded for each chunk is constructed by mode strategies in vexor/modes.py. You can adjust the strategy label construction in that file to encode additional context.
  8. Configure Vexor settings via CLI and config files

    main

    Vexor configuration is managed through the vexor config command or the interactive vexor init wizard.

    Configuration Hierarchy

    Settings are resolved using the following precedence (from lowest to highest):

    1. Built-in defaults
    2. Global configuration in ~/.vexor/config.json
    3. Project-local configuration in <project>/.vexor/config.json
    4. Environment variable overrides (e.g., VEXOR_CONFIG_JSON)
    5. Explicit runtime or Python API arguments

    Project-Local Overrides

    To use project-specific settings, create a .vexor/ directory in your project root. Vexor walks upward from the target path to find the nearest .vexor/ directory. If a config.json exists there, it is overlaid on the global config.

    Note: Mutating commands like vexor config --set-* and vexor init only modify the global config. To apply project-specific overrides, you must edit <project>/.vexor/config.json directly.

    // Example <project>/.vexor/config.json
    {
      "model": "text-embedding-3-small",
      "embedding_dimensions": 1024,
      "batch_size": 32,
      "embed_concurrency": 4,
      "extract_concurrency": 4,
      "auto_index": true,
      "rerank": "hybrid"
    }
  9. Configure Vexor for Cursor, Windsurf, and other JSON-based MCP clients

    main

    For clients that use a JSON configuration file, add the following entry to your mcpServers configuration. If vexor is not in your system PATH, use an absolute path or run via python:

    {
      "mcpServers": {
        "vexor": {
          "command": "vexor",
          "args": ["mcp"]
        }
      }
    }

    Alternative (via Python):

    {
      "mcpServers": {
        "vexor": {
          "command": "python",
          "args": ["-m", "vexor", "mcp"]
        }
      }
    }
    {
      "mcpServers": {
        "vexor": {
          "command": "vexor",
          "args": ["mcp"]
        }
      }
    }
  10. Quick Start with Vexor CLI

    main

    Vexor provides a guided setup wizard to configure your environment. After setup, you can perform semantic searches or explicitly build indexes.

    0. Guided Setup

    Run the wizard to configure providers and API keys:

    vexor init

    Search for files based on semantic meaning. Vexor auto-indexes on the first search if no index exists.

    • Search current directory:
      vexor "query"
    • Search an explicit path with result limits:
      vexor search "query" --path ~/projects/demo --top 5
    • In-memory search (no cache):
      vexor search "query" --no-cache

    2. Explicit Indexing

    Manually trigger indexing, which is useful for CI warmups or when auto_index is disabled.

    • Index current directory:
      vexor index
    • Index an explicit path with a specific mode:
      vexor index --path ~/projects/demo --mode code
    vexor init
    vexor "api client config"
    vexor index --path ~/projects/demo --mode code
  11. Manage API Keys and Environment Variables

    main

    Vexor supports multiple ways to provide credentials.

    API Keys

    You can set a key via CLI:

    vexor config --set-api-key "YOUR_KEY"

    Or use environment variables (in order of precedence):

    1. VEXOR_API_KEY (takes precedence over stored keys)
    2. OPENAI_API_KEY
    3. GOOGLE_GENAI_API_KEY
    4. VOYAGE_API_KEY

    Configuration via Environment

    • VEXOR_CONFIG_JSON: Allows injecting a JSON object of non-secret configuration fields. This is useful for CI or MCP clients. Note: Credential fields (like api_key) are rejected in this variable; use the dedicated environment variables above.
    • VEXOR_NO_UPDATE_CHECK=1: A hard switch to disable automatic update checks, overriding all other layers.
  12. Manage scoped configuration with VexorClient

    main

    Use VexorClient as a context manager to handle scoped configuration, reusable indexes, and custom data directories without mutating global state. This is recommended for library usage where you want to isolate settings like data_dir or api_key.

    from vexor import VexorClient
    
    with VexorClient(data_dir="/tmp/vexor", use_config=False) as client:
        client.set_config_json({"provider": "openai", "api_key": "YOUR_KEY"})
        response = client.search("config loader", path=".", mode="name")