Crawl4AI RAG MCP Server

The knowledge graph system uses Neo4j to store the structural components of GitHub repositories. This allows for AI hallucination detection and repository analysis.

Schema

The database maps code structure using the following nodes and relationships:

Nodes:

• Repository: GitHub repositories
• File: Python files
• Class: Python classes
• Method: Class methods (includes parameter info)
• Function: Standalone functions
• Attribute: Class attributes

Relationships:

• Repository -[:CONTAINS]-> File
• File -[:DEFINES]-> Class or Function
• Class -[:HAS_METHOD]-> Method
• Class -[:HAS_ATTRIBUTE]-> Attribute

Workflow

1. Parsing: Use the parse_github_repository tool to index a codebase (ensure the URL ends with .git).
2. Validation: Use check_ai_script_hallucinations to validate generated Python scripts against the graph.
3. Exploration: Use query_knowledge_graph to explore classes and methods.

Manual Hallucination Detection

You can manually run the detector on a script using:

python knowledge_graphs/ai_hallucination_detector.py [full path to your script to analyze]

The server supports five independent RAG strategies that can be toggled via environment variables:

1. USE_CONTEXTUAL_EMBEDDINGS: Enhances chunks with context from the full document using an LLM. Best for high-precision retrieval in technical docs. Increases indexing time and cost.
2. USE_HYBRID_SEARCH: Combines keyword and semantic vector search. Best for technical terms and exact matches. Adds slight computational overhead.
3. USE_AGENTIC_RAG: Extracts and summarizes code blocks (≥300 chars) into a dedicated vector table. Provides the search_code_examples tool. Best for AI coding assistants. Increases indexing time and cost.
4. USE_RERANKING: Uses a local cross-encoder (cross-encoder/ms-marco-MiniLM-L-6-v2) to reorder search results by relevance. Improves precision for complex queries. Adds ~100-200ms to queries.
5. USE_KNOWLEDGE_GRAPH: Uses Neo4j to parse GitHub repos and validate AI code. Provides parse_github_repository, check_ai_script_hallucinations, and query_knowledge_graph tools. Best for hallucination detection.

To connect via stdio transport, add the following to your MCP configuration. This method requires providing all necessary environment variables directly in the config.

Standard Stdio Configuration:

{
  "mcpServers": {
    "crawl4ai-rag": {
      "command": "python",
      "args": ["path/to/crawl4ai-mcp/src/crawl4ai_mcp.py"],
      "env": {
        "TRANSPORT": "stdio",
        "OPENAI_API_KEY": "your_openai_api_key",
        "SUPABASE_URL": "your_supabase_url",
        "SUPABASE_SERVICE_KEY": "your_supabase_service_key",
        "USE_KNOWLEDGE_GRAPH": "false",
        "NEO4J_URI": "bolt://localhost:7687",
        "NEO4J_USER": "neo4j",
        "NEO4J_PASSWORD": "your_neo4j_password"
      }
    }
  }
}

Docker with Stdio Configuration:

{
  "mcpServers": {
    "crawl4ai-rag": {
      "command": "docker",
      "args": ["run", "--rm", "-i", 
               "-e", "TRANSPORT", 
               "-e", "OPENAI_API_KEY", 
               "-e", "SUPABASE_URL", 
               "-e", "SUPABASE_SERVICE_KEY",
               "-e", "USE_KNOWLEDGE_GRAPH",
               "-e", "NEO4J_URI",
               "-e", "NEO4J_USER",
               "-e", "NEO4J_PASSWORD",
               "mcp/crawl4ai"],
      "env": {
        "TRANSPORT": "stdio",
        "OPENAI_API_KEY": "your_openai_api_key",
        "SUPABASE_URL": "your_supabase_url",
        "SUPABASE_SERVICE_KEY": "your_supabase_service_key",
        "USE_KNOWLEDGE_GRAPH": "false",
        "NEO4J_URI": "bolt://localhost:7687",
        "NEO4J_USER": "neo4j",
        "NEO4J_PASSWORD": "your_neo4j_password"
      }
    }
  }
}

If you prefer running the server directly without Docker, use uv to manage the Python environment. Note that for Knowledge Graph functionality, running via uv is currently recommended over Docker.

1. Clone the repository:

   git clone https://github.com/coleam00/mcp-crawl4ai-rag.git
   cd mcp-crawl4ai-rag

2. Install uv if not already present:

   pip install uv

3. Create and activate a virtual environment:

   uv venv
   # Windows:
   .venv\Scripts\activate
   # Mac/Linux:
   source .venv/bin/activate

4. Install dependencies and setup Crawl4AI:

   uv pip install -e .
   crawl4ai-setup

5. Create a .env file based on the project's configuration requirements.

git clone https://github.com/coleam00/mcp-crawl4ai-rag.git
cd mcp-crawl4ai-rag
pip install uv
uv venv
# on Mac/Linux: source .venv/bin/activate
uv pip install -e .
crawl4ai-setup

The server is configured via a .env file in the project root. Key configuration categories include:

• MCP Server: HOST, PORT, and TRANSPORT (e.g., sse or stdio).
• OpenAI: OPENAI_API_KEY and MODEL_CHOICE (the LLM used for summaries and embeddings).
• RAG Strategies: Boolean flags to enable specific retrieval behaviors (USE_CONTEXTUAL_EMBEDDINGS, USE_HYBRID_SEARCH, USE_AGENTIC_RAG, USE_RERANKING, USE_KNOWLEDGE_GRAPH).
• Supabase: SUPABASE_URL and SUPABASE_SERVICE_KEY for vector storage.
• Neo4j: NEO4J_URI, NEO4J_USER, and NEO4J_PASSWORD (required if USE_KNOWLEDGE_GRAPH=true).

# MCP Server Configuration
HOST=0.0.0.0
PORT=8051
TRANSPORT=sse

# OpenAI API Configuration
OPENAI_API_KEY=your_openai_api_key
MODEL_CHOICE=gpt-4.1-nano

# RAG Strategies
USE_CONTEXTUAL_EMBEDDINGS=false
USE_HYBRID_SEARCH=false
USE_AGENTIC_RAG=false
USE_RERANKING=false
USE_KNOWLEDGE_GRAPH=false

# Supabase Configuration
SUPABASE_URL=your_supabase_project_url
SUPABASE_SERVICE_KEY=your_supabase_service_key

# Neo4j Configuration
NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=your_neo4j_password