DOCSEARCHbyAlgolia
Submit Docs

Lumen Semantic Code Search

repository·main·Indexed 42 days ago

https://github.com/ory/lumen

A 100% local semantic code search engine designed for AI coding agents. Lumen acts as an MCP server that uses local embedding models (Ollama/LM Studio) and SQLite with sqlite-vec to provide privacy-preserving code retrieval. It supports semantic chunking for 12 language families, including Go, Python, Rust, and TypeScript, and integrates with agents such as Claude Code, Cursor, Codex, and OpenCode.

Tokens
18.5K
Snippets
50
Records
94
Agent score
55%

What's inside Lumen

  1. Analyze Lumen benchmark performance and cost

    main

    Lumen's performance can be evaluated by comparing it against a baseline using metrics such as Total Cost, Avg Time, Avg Output Tokens, and Search Hit Rate.

    Key performance characteristics observed in benchmarks:

    • Cost/Time Efficiency: In most tasks, using Lumen is cheaper and faster than the baseline. For example, in JavaScript tasks, cost can decrease by ~32% and time by ~53%.
    • Search Hit Rates: Search results are classified as DIRECT_HIT, FILE_HIT, MISS, or ERROR.
    • Regression Scenarios: Costs may increase in 'green-field' tasks where the agent is writing new files that do not yet exist in the index, or when search results are large enough to trigger tool result limits.

  2. Lumen benchmark results overview

    main

    Lumen benchmarks across 9 languages show that while quality (patch rating) is maintained, Lumen consistently reduces operational costs.

    Summary for Bug-Fix Tasks (excluding C++ feature task):

    • Cost: Reduced by ~37% ($0.43 $\rightarrow$ $0.27 avg).
    • Time: Reduced by ~37% (183s $\rightarrow$ 116s avg).
    • Output tokens: Reduced by ~42% (8,278 $\rightarrow$ 4,787 avg).

    Summary for All 9 Tasks (including C++ feature task):

    • Cost: Reduced by ~26% ($0.50 $\rightarrow$ $0.37 avg).
    • Time: Reduced by ~28% (204s $\rightarrow$ 146s avg).
    • Output tokens: Reduced by ~25% (9,439 $\rightarrow$ 7,042 avg).

  3. How Lumen works: Semantic indexing and search

    main

    Lumen acts as an MCP (Model Context Protocol) server between your codebase and your AI agent.

    The Indexing Pipeline:

    1. File Walking & Merkle Tree: On session start, Lumen walks the project and builds a Merkle tree over file hashes. This allows for efficient incremental updates by only re-processing changed files.
    2. Semantic Chunking: Files are split into semantic chunks (functions, types, methods) using Go's native AST or tree-sitter grammars.
    3. Embedding: Chunks are converted into vector embeddings via a local server (Ollama or LM Studio).
    4. Storage: Embeddings are stored in SQLite + sqlite-vec using cosine-distance KNN for retrieval.

    Data Flow: Files $\rightarrow$ semantic chunks $\rightarrow$ vector embeddings $\rightarrow$ SQLite/sqlite-vec $\rightarrow$ KNN search

    Storage Location: The index is stored outside your repository at ~/.local/share/lumen/<hash>/index.db. Git worktrees from the same repository automatically share index data if they use a compatible model, vector-storage, and chunking profile.

  4. Understand why test cases are invisible to search

    main

    In JavaScript and TypeScript projects, test blocks such as describe(), it(), and test() may not be indexed by the semantic search.

    Reason: Tree-sitter queries used for indexing currently only capture function_declaration and class_declaration. Because test frameworks use these function calls to define test suites rather than standard declarations, they do not produce searchable chunks.

  5. Classify search call results

    main

    Lumen search calls are classified into several categories to evaluate performance and accuracy. When analyzing search hits, use the following classifications:

    • DIRECT_HIT: The search correctly identifies the gold file and the specific gold symbol.
    • FILE_HIT: The search identifies the correct file, but not necessarily the specific symbol, or identifies a related file.
    • MISS: The search fails to find the relevant files (often because it is limited to existing files).
    • ERROR: The search call failed due to technical issues, such as an MCP validation error (e.g., invalid n_results) or a Result overflow (output exceeding character limits).

  6. Understand Chunker Analysis metrics

    main

    Chunker Analysis reports how effectively the Lumen chunking strategy retrieves relevant files for specific programming languages during benchmark tasks. The following metrics are used to evaluate performance:

    • Tasks: The number of benchmark tasks assigned for the language.
    • Runs: The number of search executions performed (split into valid and invalid runs).
    • Queries: The number of search queries issued.
    • Hit Rate: The percentage of 'gold-patch' files (files known to be relevant to the task) successfully found by the search.
    • Noise Rate: The percentage of search results that were NOT part of the gold patch (irrelevant results).

  7. Understand bench-swe metrics and results

    main

    Each task in a benchmark run produces several metrics and output files.

    Metrics tracked per task/scenario

    • Cost (USD): Derived from the Claude API usage.
    • Duration (ms): Time taken to complete the task.
    • Tokens: Breakdown of input, cache read, cache created, and output tokens.
    • Quality: A rating of Poor / Good / Perfect judged by Opus 4.6.

    Benchmark Scenarios

    • baseline: Uses all built-in tools (Read, Edit, Write, Grep, etc.) but no MCP tools.
    • with-lumen: Uses all built-in tools plus Lumen MCP tools (semantic_search, index_status).

    Result Directory Structure

    Results are stored in bench-results/swe-YYYYMMDD-HHMMSS-backend-model/ and include:

    • *-raw.jsonl: Full Claude stream output.
    • *-patch.diff: The patch generated by Claude.
    • *-metrics.json: Cost, time, and token data.
    • *-judge.json: The quality rating.
    • *-judge.md: Explanation for the quality rating.
    • summary-report.md: Aggregate tables.
    • detail-report.md: Full patches and analysis.
    • chunker-analysis.md: Chunker effectiveness report.

  8. Handle oversized chunks and symbol name merging

    main

    When content exceeds LUMEN_MAX_CHUNK_TOKENS * 4 characters, splitOversizedChunks in internal/index/split.go splits at line boundaries. Currently, mergeUndersizedChunks concatenates adjacent small chunks using a + in the symbol name (e.g., symbolA+symbolB).

    Recommendations for improvement:

    1. Preserve Names: For sub-chunks created by splitting, preserve the original symbol name instead of combining them with +.
    2. Boundary Detection: Implement inner-block boundary detection for very large methods to avoid imprecise chunk boundaries.

  9. Understand Lumen index storage and collection identity

    main

    Lumen stores indexes in repository-scoped, content-addressed SQLite collections located outside your source repository. This allows multiple Git worktrees to share unchanged file revisions and embeddings, saving disk space and time.

    Storage Location

    The default path is: ~/.local/share/lumen/<profile-hash>/index.db (Note: XDG_DATA_HOME replaces ~/.local/share if set).

    Collection Identity (The Profile Hash)

    A collection is uniquely identified by a profile hash derived from:

    • The resolved Git common directory (or absolute path for non-Git projects).
    • The indexed scope.
    • Embedding model and vector dimensions.
    • Vector storage type (int8 or float32).
    • Maximum chunk size.
    • Lumen's index-format version.

    Important: Changing the embedding model, dimensions, vector precision, chunk size, or format version will automatically create a new, separate collection. The embedding backend (e.g., Ollama vs. LM Studio) is not part of the profile hash; if using different backends with the same model name, use distinct configured model names to avoid conflicts.

  10. Understand the Lumen evaluation methodology

    main

    Lumen is evaluated using bench-swe, a SWE-bench-style harness designed to measure how Lumen impacts the cost, time, and token usage of AI agents (specifically Claude) when fixing real GitHub bugs.

    The evaluation compares two scenarios:

    1. baseline: Claude using only default tools (Read, Write, Edit, Grep, Bash, etc.) without Lumen.
    2. with-lumen: Claude using default tools plus Lumen's semantic_search MCP tool.

    Each task involves a real GitHub bug from an open-source project. The agent is provided with the issue description and the codebase at the pre-fix commit and must produce a patch to fix the issue.

  11. How Lumen patches are judged

    main

    Patches generated during benchmarks are evaluated by Claude Sonnet 4.6 acting as a blind judge. The judge compares the generated patch against a known-correct 'gold patch' using the following criteria:

    • Perfect: The patch fixes the issue with logic that is equivalent to or better than the gold patch.
    • Good: The patch fixes the issue correctly but uses a different valid approach.
    • Poor: The patch is incorrect, incomplete, fails to compile, or does not fix the issue.

    Additionally, the judge evaluates:

    • files_correct: Whether the patch modified the appropriate files.
    • logic_equivalent: Whether the fix is semantically identical to the gold patch.

  12. Build Lumen from source

    main

    To develop on Lumen locally, you must have CGO enabled (required for sqlite-vec).

    git clone https://github.com/ory/lumen.git
cd lumen

# Build locally
make build-local

# Run tests
make test

# Run linter
make lint

# Load as a Claude Code plugin from source
make plugin-dev
    git clone https://github.com/ory/lumen.git
cd lumen
make build-local
make test
make lint
make plugin-dev