DOCSEARCHbyAlgolia
Submit Docs

jDocMunch MCP Server

repository·master·Indexed 41 days ago

https://github.com/jgravelle/jdocmunch-mcp

A token-efficient Model Context Protocol (MCP) server for coding agents that enables structured documentation retrieval via section-level indexing. It uses heading hierarchy and byte-precise extraction to minimize context window usage and token costs. Supports local-first indexing of Markdown, RST, HTML, OpenAPI, and other formats, with optional hybrid semantic search via embedding providers like FastEmbed, OpenAI, or Gemini.

Tokens
55K
Snippets
144
Records
243
Agent score
52%

What's inside jDocMunch

  1. Overview of jDocMunch MCP capabilities

    master

    jDocMunch provides 64 tools categorized into several functional areas to help agents interact with documentation efficiently:

    • Precise Retrieval: Use get_section, get_sections, and get_section_excerpt to pull exact content from files based on byte offsets.
    • Semantic & Keyword Search: Use search_sections to find content by meaning. Supports hybrid search (BM25 + semantic) if an embedding provider is configured. Options like compact=true and snippet_bytes=N can reduce response size.
    • Structural Navigation: Traverse the heading tree without loading content using get_toc, get_toc_tree, get_section_path, get_section_descendants, and section_neighbors.
    • Documentation Health: Audit your docs using get_doc_coverage, get_undocumented_symbols, get_stale_pages, get_orphan_sections, get_broken_links, and doc_health_radar.
    • API Specification Support: Treat OpenAPI documents as first-class citizens with find_endpoint, list_endpoints_by_tag, find_operations_using_schema, and get_schema_graph.
    • Change Impact Analysis: Use check_section_delete_safe and get_section_blast_radius to evaluate the impact of restructuring or deleting sections.
    • Freshness Metadata: Content responses include _meta.freshness and _meta.verdict to indicate how current the information is.

  2. Use claim-scoped evidence in handoff v2

    master

    Starting with v1.116.0, the handoff protocol supports v2 which allows for claim-scoped evidence. Instead of a single global block of references at the end of a document, each claim can now carry its own evidence_refs.

    Key behaviors:

    • Claim Identification: Claims use ids that must be unique across the entire handoff, not just within a section. The id serves as the citation anchor.
    • Verbatim Preservation: Statements and classifications are preserved verbatim; the server does not author them.
    • Validation: The _validate_claims method accepts {id, statement, evidence_refs, classification?}. If a reference is unknown, it returns invalid_claims: [{claim_id, unknown_refs}] specifically naming the problematic claim.
    • Schema Promotion: If no claims are present, the schema remains jdocmunch.handoff/v1. The presence of any claim promotes the handoff to .../v2.
    • Rendering: render_handoff prints the statement as a header (### <statement>) followed by indented evidence and the claim ID.

    Known Limitation (Phase 1):

    Phase 1 does not narrow what counts as a match. Citing a whole document may still attest to a claim if even one unrelated section from that document was served. (Phase 2 'evidence receipts' is deferred).

    // Example of the v2 claim structure concept
{
  "schema": "jdocmunch.handoff/v2",
  "claims": [
    {
      "id": "unique_anchor_1",
      "statement": "The text says X.",
      "evidence_refs": ["ref_abc"]
    }
  ]
}

  3. Understand corpus exclusion and inheritance

    master

    The tool manages which files are included in an index using corpus_shape_patterns.

    • Inheritance: If no specific patterns are provided (represented as None), the tool inherits the existing patterns from the stored index.
    • Explicit Empty: Providing an empty list [] is treated as an explicit instruction to include everything (widening the scope) and is recorded as such.
    • Persistence: Patterns are persisted via corpus_shape_patterns across all five persistence paths to ensure that incremental refreshes and watchers respect the same exclusion rules as the initial full index.
    • Watcher Behavior: The watch process respects both the source root's .gitignore and the stored corpus_shape_patterns to prevent ignored files from being re-added during filesystem events.

  4. Identify read-only vs mutating MCP tools

    master

    jDocMunch MCP tools are annotated with readOnlyHint to allow clients (like Claude Code) to distinguish between safe queries and operations that mutate persistent state.

    Non-Read-Only Tools (Mutating):

    • index_local
    • doc_index_repo
    • delete_index
    • define_repo_group
    • tune_weights
    • check_embedding_drift (Note: check_embedding_drift is marked as mutating because force=true re-pins the canary).

    Read-Only Tools (Non-Mutating):

    • link_code_to_symbols
    • verify_index
    • resolve_related_code_repos
    • All standard query tools.

  5. Understand embedding rotation and paid provider behavior

    master

    An embedding_rotation is disclosed when the system detects a mismatch in embedding identity (e.g., a model change or a change in character limit).

    Paid Providers (openai/gemini): To prevent unexpected costs and undisclosed egress, the system never auto-escalates for paid providers. Instead:

    1. It returns action: "rebuild_required".
    2. It keeps the existing vectors.
    3. It allows search to degrade while disclosing the rotation status.

    Consent: The system reuses the existing JDOCMUNCH_ALLOW_PAID_EMBEDDINGS setting for these operations; no new consent knob is required.

  6. Understand the difference between durable and per-call settings

    master

    When configuring the project, be aware of how settings persist:

    • Durable Settings: Settings like corpus_shape_patterns (used by index-local exclusions) are stored in a manifest and persist across sessions.
    • Per-call Arguments: Settings like use_ai_summaries are passed during the function call and are not stored in a manifest.

    Warning: A per-call setting will not be honored by background processes (like the watcher) unless specifically passed to them, as they do not read from the persistent manifest.

  7. Understand the BM25 tokenizer behavior for Unicode and CJK

    master

    As of v1.114.1, the BM25 tokenizer in retrieval/tokenize.py supports Unicode word boundaries and CJK (Chinese, Japanese, Korean) languages.

    • Unicode/Latin: Uses [\W_]+ for splitting, correctly handling accented characters (e.g., café).
    • CJK: Uses _CJK_RE to identify Hangul, Hiragana, Katakana, and Han characters. It pads runs with spaces before splitting and expands them into overlapping character bigrams via _cjk_bigrams.
    • Matching: Bigram expansion occurs at both index and query time to ensure matches.
    • Note: This change does not require a reindex, as BM25/prune/dedup tokenize content at load/scoring time. The .terms.json glossary remains untouched.

  8. How jDocMunch works: Architecture and Retrieval

    master

    jDocMunch operates as a local-first MCP server. All indexing and retrieval happen on your machine; no hosted service is required.

    Data Flow

    docs/ $\rightarrow$ parser $\rightarrow$ section tree $\rightarrow$ local index $\rightarrow$ MCP client $\leftarrow$ retrieval

    Key Components

    • Parsing: Format-specific modules support Markdown/MDX, reStructuredText, AsciiDoc, Jupyter notebooks, HTML, plain text, OpenAPI (YAML), JSON/JSONC, XML/SVG/XHTML, Godot scenes, and (via [office] extra) PDF, DOCX, PPTX, and EPUB.
    • Storage: Uses a versioned local index (INDEX_VERSION = 3) that supports automatic migration. 1.x releases are compatible and do not force re-indexing.
    • Retrieval:
      • Lexical: Uses BM25 by default.
      • Hybrid: Combines BM25 with semantic search when an embedding provider is configured.
    • Embeddings: Optional and provider-agnostic. Supports Gemini, OpenAI, OpenAI-compatible endpoints, or local models via FastEmbed (ONNX) or sentence-transformers (torch). If no provider is configured, search remains entirely offline and lexical.

  9. Review index coverage metadata

    master

    To prevent false 'absent' verdicts caused by files being skipped during indexing, the project persists DocIndex.coverage metadata during full discovery walks.

    When a retrieval verdict is absent or low_confidence, the coverage block may be attached to the response. This block includes:

    • generation: {indexed_at, index_version, git_head}
    • files_indexed: Total count of files indexed.
    • excluded: A map of {reason: count} (e.g., files skipped due to parse errors or specific skip reasons).
    • no_sections_files: Count of files that were indexed but contained no valid sections.

  10. Understand jDocMunch retrieval capabilities and behaviors

    master

    jDocMunch's semantic section indexing enables several advanced retrieval behaviors that go beyond simple keyword matching:

    • Byte-precise extraction: It can extract small, specific sections from very large files (e.g., extracting an 863-byte section from a 95 KB file) without loading the surrounding waste.
    • Automatic ecosystem discovery: Semantic indexing can connect queries to related subdirectories or providers (e.g., discovering CNI providers) even if the user doesn't explicitly mention them.
    • Multi-tier documentation coverage: The system naturally traverses different documentation layers, including Concepts, Tasks, Reference, and Tutorials.
    • Implicit cross-referencing: It can surface relevant migration paths (e.g., dockershim to CRI) even when the user only asks a generic question about the new technology.

  11. Understand tool tiers (core vs standard) as scope choices

    master

    The tool tiers (core, standard) in jDocMunch MCP are scope choices rather than token-saving levers.

    When selecting a tier, understand that it defines which tools are included in the surface, but it does not function as a direct mechanism to reduce token counts in the way a user might expect from the configuration surface.

    Measured impact at 64 tools / 13,252 schema tokens:

    • core: ~ -62.08% (drops 49 tools)
    • standard: ~ -9.39% (drops 8 tools)

  12. How jDocMunch-MCP optimizes token usage

    master

    jDocMunch-MCP achieves high token efficiency by indexing documentation into discrete sections and using AI-generated summaries. Instead of the 'Naive Approach' where an LLM must read entire files (potentially hundreds of thousands of tokens), jDocMunch uses semantic search to identify the exact relevant sections.

    When a user performs a search or retrieval, only the specific section content and metadata are returned. This results in significantly lower token consumption (often ~70x fewer tokens) and avoids the high cost of loading large files into an LLM's context window.