DOCSEARCHbyAlgolia
Submit Docs

Grounded Docs MCP Server

repository·main·Indexed 23 days ago

https://github.com/arabold/docs-mcp-server

An MCP server and CLI tool designed to provide AI coding assistants with version-specific documentation. It indexes websites, GitHub, npm, PyPI, and local files to reduce hallucinations. Features include semantic vector search via embedding models, Markdown-optimized scraping with llms.txt support, and a modular pipeline for processing diverse file formats including PDF, Office, and source code. Supports both unified and distributed deployment modes via stdio or HTTP/SSE transports.

Tokens
84.8K
Snippets
112
Records
449
Agent score
81%

What's inside @arabold/docs-mcp-server

  1. How dimension detection is performed

    main

    The system determines the effective embedding dimension during initialization. To optimize performance and avoid unnecessary API calls, it follows these rules:

    • Skip Probing (Known Models): If the configured model has a known fixed output (e.g., openai:text-embedding-3-small), the system uses the known dimension and skips the startup probe.
    • Skip Probing (Matching Metadata): If the current embedding_model matches stored metadata that already contains an embedding_dimension, the system reuses that dimension and skips the probe.
    • Runtime Probing: If the model is unknown/variable and no matching metadata or explicit override exists, the system performs a startup probe by generating an embedding for the string "test" and measuring its length.
    • Explicit Overrides: If embeddings.vectorDimension is explicitly configured, the system uses that value immediately and skips all auto-detection/probing logic.

  2. Understand how baseline files are named and stored

    main

    Baseline files are derived deterministically from the dataset path and the provider name to prevent accidental overwrites and ensure comparability. The naming convention follows these rules:

    • Canonical dataset + local provider: tests/search-eval/baseline.json (the default).
    • Canonical dataset + non-local provider: tests/search-eval/baseline.<provider>.json (e.g., baseline.context7.json).
    • Non-canonical dataset: A sibling file named <dataset-stem>[.provider].baseline.json.

  3. Understand the Job Lifecycle and States

    main

    Jobs move through a specific set of states during their execution. Understanding these states is critical for monitoring and error handling.

    Job States:

    • QUEUED: Job is created and waiting in the queue for an available worker.
    • RUNNING: A PipelineWorker is currently processing the job.
    • COMPLETED: The job finished successfully.
    • FAILED: An error occurred during processing.
    • CANCELLED: The job was manually stopped by a user.

    State Transition Flow: QUEUED $\rightarrow$ RUNNING $\rightarrow$ COMPLETED (or FAILED / CANCELLED)

  4. Exclude structural chunks from search results

    main

    To prevent container elements (like function signatures or class declarations) from appearing as standalone search results, the system automatically filters them out.

    Any chunk whose metadata.types array contains the string 'structural' is excluded from all search results, regardless of whether you are using Hybrid Search or FTS-only mode.

  5. Interpreting command output and diagnostics

    main

    All docs-search commands emit structured data to stdout.

    • Programmatic access: Parse stdout as JSON (the default) or use --output yaml for human-readable structured data.
    • Diagnostics/Progress: These are sent to stderr. They are suppressed by default in non-interactive sessions.
    • Re-enabling diagnostics: Use the --verbose flag or set the environment variable LOG_LEVEL=INFO.
    • Suppressing diagnostics: Use the --quiet flag to suppress all non-error diagnostics regardless of the session type.

  6. Understand `docs-manage` command output behavior

    main

    The docs-manage commands (scrape, refresh, remove) follow a specific output pattern:

    • stdout: Receives plain-text status messages (e.g., Successfully scraped 42 pages).
    • stderr: Receives progress updates and diagnostics.

    Controlling Verbosity:

    • In non-interactive sessions, diagnostics are suppressed by default.
    • Use --verbose or set the environment variable LOG_LEVEL=INFO to enable debug logging.
    • Use --quiet to suppress all non-error diagnostics regardless of the session type.
    • Note: The --output flag is accepted but has no effect as output is always plain text.

  7. How size management affects chunking

    main

    To prevent chunks from becoming too large for LLM context windows, the splitter enforces a maximum size (bytes or tokens).

    The Process:

    1. The splitter first identifies semantic boundaries (e.g., the start and end of a function).
    2. If a semantic unit (like a large function body) exceeds the maximum size, it is delegated to a TextSplitter.
    3. The TextSplitter breaks the large content into smaller sub-chunks.
    4. Guarantees:
      • The first sub-chunk of a large declaration retains the signature and documentation.
      • Sub-chunks inherit the parent's hierarchical path.
      • Sub-chunks are classified as boundaryType: "content".
      • The original order and byte integrity are preserved for perfect reconstructability.

  8. How to handle hash-routed SPAs with `preserveHashes`

    main

    Use scraper.preserveHashes for documentation sites that use hash-based SPA routes (e.g., https://docs.example.com/#/guide). For normal sites, leave it disabled as hashes usually point to anchors on the same page.

    Usage across interfaces:

    • CLI scrape: Use --preserve-hashes to enable hash-aware crawling.
    • CLI refresh: Use --preserve-hashes to override the stored setting.
    • MCP: The scrape_docs tool accepts preserveHashes: true.
    • Web UI: Use the "Preserve Hash Routes" checkbox in the scrape form.

    Note: If preserveHashes is enabled and scrapeMode is fetch, the job is automatically upgraded to use playwright.

  9. Dataset Requirements for Search Evaluation

    main

    To run a valid benchmark, the dataset must meet the following criteria:

    • Format: A checked-in dataset of (library, query, qrels) tuples.
    • Graded Relevance: qrels must contain one or more document URLs, each with an integer grade ≥ 1 (higher = more relevant). Binary relevance is insufficient; at least one query must use multiple distinct grade values.
    • Coverage: At least five distinct libraries and at least fifty queries in total.
    • Diversity: Queries must include a mix of intents (e.g., api-lookup, conceptual, comparison, troubleshooting).
    • Library Presence: All libraries referenced in the dataset must be indexed in the store before the run starts. If a library is missing, the benchmark will fail fast and provide the necessary scrape command.

  10. Use hierarchy for advanced search and retrieval

    main

    The level and path properties enable sophisticated database operations for context-aware search:

    • Parent chunks: Retrieve using path.slice(0, -1).
    • Child chunks: Find paths that start with the current path plus one additional element.
    • Sibling chunks: Find chunks with the same path length and a shared parent.
    • Search Context: When a chunk matches a search, the system can automatically provide:
      1. Direct match: The specific chunk.
      2. Parent context: Broader context for understanding.
      3. Sibling navigation: Related content at the same level.
      4. Child exploration: Deeper content for more details.

  11. How embedding dimensions are resolved in the store

    main

    The store determines the effective vector dimension used for the documents_vec SQLite table and embedding validation by following a specific hierarchy of precedence. This ensures that the database schema and stored vectors remain consistent even when using OpenAI-compatible providers or resizable (Matryoshka) models.

    Dimension Resolution Hierarchy

    1. Explicit Override: If you set embeddings.vectorDimension via configuration, environment variables, or the CLI, this value is used immediately. The model output is then validated against this dimension.
    2. Stored Metadata: If the embedding_model in your current configuration matches the embedding_model stored in the database metadata, the store uses the previously detected embedding_dimension. This avoids unnecessary paid API calls to the provider during startup.
    3. Runtime Detection (Startup Probe): For unknown models or variable-dimension models where no metadata exists, the server performs a startup probe by embedding the string "test". The resulting dimension is then persisted as embedding_dimension.
    4. Known-Dimension Lookup: For stable, fixed-output models (e.g., OpenAI, Vertex, Bedrock, Cohere, Voyage, or specific Hugging Face models), the store uses a pre-defined lookup table to avoid probing.

    Key Behaviors

    • Resizable Models: Models known to support multiple dimensions bypass the known-dimension lookup and are treated as runtime-detected to ensure the dimension matches the provider's current configuration.
    • Dimension Validation: If an explicit override is provided, the store validates the model's output against it. If the dimensions mismatch, a DimensionError is raised (unless the embedding wrapper explicitly supports Matryoshka/MRL truncation).

  12. Markdown preference logic for llms.txt and standard crawls

    main

    The scraper follows a specific hierarchy when determining which content to process, depending on how the URL was discovered:

    For items discovered via llms.txt:

    1. Implicit .md variant: The scraper first attempts to fetch the URL with a .md extension. If this succeeds and is valid Markdown, it is used.
    2. HTML Markdown Alternate: If the implicit .md fetch fails, the scraper fetches the original HTML and looks for <link rel="alternate" ...> tags. If a valid Markdown alternate is found, it is used.
    3. Standard HTML: If neither the implicit .md nor the HTML alternate is available, the scraper processes the original HTML.

    For standard queue items (not from llms.txt):

    1. Markdown-preferred Accept: The scraper fetches the URL using standard Accept headers to request Markdown if available.
    2. HTML Markdown Alternate: If the response is HTML, the scraper then inspects the HTML for <link rel="alternate" ...> tags to find a Markdown version.