Open Deep Research MCP

repository·main·Indexed 43 days ago

https://github.com/teelaitila/deep-research-mcp

An AI-powered research assistant that performs iterative research using search engines and web scraping. It functions as a standalone CLI or a Model Context Protocol (MCP) server for AI agents like Claude Desktop. It supports multiple LLM providers including OpenAI, Anthropic, Google, and xAI, and integrates with Firecrawl for web scraping and Langfuse for observability.

Tokens
5.6K
Snippets
15
Records
25
Agent score
50%

What's inside Open Deep Research

  1. Understand the output format of research reports

    main

    The deep-research-mcp server generates comprehensive research reports that include structured analysis and source evaluation. Each source cited in a report is accompanied by a Reliability score (typically on a scale from 0.0 to 1.0) and a qualitative justification. This allows users to assess the credibility of the information provided in the final report.

    Example source entry structure:

    • URL: The direct link to the source.
    • Reliability: A numerical score (e.g., 0.60) representing the assessed trustworthiness.
    • Justification: A prose explanation of why that specific score was assigned (e.g., noting commercial bias, lack of editorial oversight, or domain expertise).
    • Title: The title of the webpage or document.
  2. Understand the output format of research results

    main

    The exampleout.md file demonstrates the structure of the research output generated by the deep-research-mcp server. Each research finding is presented as a list item containing:

    1. URL: The direct link to the source material.
    2. Reliability Score: A numerical value (e.g., 0.85, 0.80) representing the assessed trustworthiness of the source.
    3. Reliability Justification: A qualitative explanation of why the source was assigned that score, considering factors like editorial standards, domain expertise, commercial bias, and peer-review status.
    4. Title: The title of the specific article or document found.

    This structured data allows users to evaluate the credibility of the information used to compile the final report.

  3. How to select models in CLI vs MCP/HTTP

    main

    Model selection behavior depends on how you are interacting with the server:

    • CLI Mode: You will be prompted interactively for the provider and model (e.g., entering openai and then gpt-5.2).
    • MCP/HTTP Mode: Pass the model directly using the format provider:model or provider/model (e.g., openai:gpt-5.2).
  4. Configure Local Firecrawl for Free Web Scraping

    main

    To avoid using the Firecrawl API, you can run a local instance of Firecrawl. This is a free option that allows you to use a local scraping backend.

    1. Clone and set up a local Firecrawl instance (e.g., using the localfirecrawl fork which uses searXNG as the search backend).
    2. Update your .env.local to point to the local instance:
    FIRECRAWL_BASE_URL="http://localhost:3002"
  5. Quick Start: Install and Run the Open Deep Research MCP Server

    main

    To get started with the Open Deep Research MCP Server, clone the repository, install dependencies, set up your environment variables, and build the project.

    1. Clone and Install

    git clone https://github.com/Ozamatash/deep-research
    cd deep-research
    npm install

    2. Configure Environment

    Copy the example environment file to .env.local to manage your API keys and configurations.

    cp .env.example .env.local

    3. Build and Run

    Build the server using npm:

    npm run build

    To run the CLI version:

    npm run start

    To run as a remote server via Streamable HTTP (runs on http://localhost:3000/mcp without session management):

    npm run start:http
    git clone https://github.com/Ozamatash/deep-research
    cd deep-research
    npm install
    cp .env.example .env.local
    npm run build
    npm run start
  6. Configure Model Providers and API Keys

    main

    The server supports OpenAI, Anthropic, Google, and xAI. You must provide the corresponding API keys in your .env.local file.

    Required API Keys

    Set the following keys in .env.local:

    • OPENAI_API_KEY
    • ANTHROPIC_API_KEY
    • GOOGLE_API_KEY
    • XAI_API_KEY

    Default Model Selection

    You can optionally define default models for each provider in .env.local:

    • OPENAI_MODEL (e.g., gpt-5.2)
    • ANTHROPIC_MODEL (e.g., claude-opus-4-5)
    • GOOGLE_MODEL (e.g., gemini-3-pro-preview)
    • XAI_MODEL (e.g., grok-4-1-fast-reasoning)

    Custom OpenAI Endpoints

    If using a non-default OpenAI endpoint, set:

    • OPENAI_ENDPOINT (e.g., https://api.openai.com/v1)
    OPENAI_API_KEY=...
    ANTHROPIC_API_KEY=...
    GOOGLE_API_KEY=...
    XAI_API_KEY=...
    
    OPENAI_MODEL=gpt-5.2
    ANTHROPIC_MODEL=claude-opus-4-5
    GOOGLE_MODEL=gemini-3-pro-preview
    XAI_MODEL=grok-4-1-fast-reasoning
    
    OPENAI_ENDPOINT=https://api.openai.com/v1
  7. Configure token budgets for research sessions

    main

    To prevent runaway costs during deep research, you can pass a tokenBudget (number) or a budgetState object to deepResearch. The engine tracks usedTokens and sets a reached: true flag when the limit is hit, causing the engine to stop further recursive steps and return the current findings.

    BudgetState structure:

    • tokenBudget?: number
    • usedTokens: number
    • reached: boolean
  8. Enable Observability with Langfuse

    main

    You can track research flows, queries, and results by integrating Langfuse. If no keys are provided, the application functions normally without observability.

    Add the following to your .env.local:

    • LANGFUSE_PUBLIC_KEY
    • LANGFUSE_SECRET_KEY
    LANGFUSE_PUBLIC_KEY="your_langfuse_public_key"
    LANGFUSE_SECRET_KEY="your_langfuse_secret_key"
  9. Run the Deep Research MCP server using Docker Compose

    main

    You can run the deep-research service using Docker Compose. The configuration expects an environment file named .env.local to be present in the root directory to provide necessary credentials and configurations. The service builds from the current directory and mounts the local project directory to /app/ inside the container to allow for live development/file access.

    services:
      deep-research:
        container_name: deep-research
        build: .
        env_file:
          - .env.local
        volumes:
          -  ./:/app/
        tty: true
        stdin_open: true
  10. Configure model selection via environment variables

    main

    You can control which models are used by the server by setting the following environment variables:

    VariableDescription
    OPENAI_MODELThe model name to use when the openai provider is selected (default: gpt-5.2)
    ANTHROPIC_MODELThe model name to use when the anthropic provider is selected (default: claude-opus-4-5)
    GOOGLE_MODELThe model name to use when the google provider is selected (default: gemini-3-pro-preview)
    XAI_MODELThe model name to use when the xai provider is selected (default: grok-4-1-fast-reasoning)
    OPENAI_ENDPOINTCustom base URL for OpenAI API (default: https://api.openai.com/v1)
    OPENAI_API_KEYRequired for OpenAI
    ANTHROPIC_API_KEYRequired for Anthropic
    GOOGLE_API_KEYRequired for Google
    XAI_API_KEYRequired for xAI
  11. Configure environment variables for Open Deep Research MCP

    main

    The server uses environment variables for configuration, which are loaded from a .env.local file located in the project root. The following environment variables are supported for configuring LLM providers, web crawling, and observability:

    LLM Providers

    • OPENAI_API_KEY: OpenAI API key.
    • OPENAI_ENDPOINT: Custom OpenAI endpoint URL.
    • ANTHROPIC_API_KEY: Anthropic API key.
    • GOOGLE_API_KEY: Google API key.
    • XAI_API_KEY: xAI API key.

    Firecrawl (Web Crawling)

    • FIRECRAWL_BASE_URL: The base URL for the Firecrawl instance. Providing this value enables isLocalFirecrawl mode (useful for local Firecrawl instances where an API key is not required).
    • FIRECRAWL_KEY: Firecrawl API key (required if FIRECRAWL_BASE_URL is not set).
    • FIRECRAWL_CONCURRENCY: Number of concurrent requests. Defaults to 2.

    Langfuse (Observability)

    • LANGFUSE_PUBLIC_KEY: Langfuse public API key.
    • LANGFUSE_SECRET_KEY: Langfuse secret API key.
    # Example .env.local content
    OPENAI_API_KEY=sk-...
    FIRECRAWL_KEY=fc-...
    LANGFUSE_PUBLIC_KEY=pk-...
    LANGFUSE_SECRET_KEY=sk-...
  12. Filter search results with filterSearchResults

    main

    The filterSearchResults function uses an LLM to evaluate a list of search results and decide which URLs are worth scraping based on a query and user preferences. It filters out SEO spam, clickbait, and irrelevant sites while respecting sourcePreferences.

    Parameters

    • searchResults: An array of objects containing url, and optionally title and description.
    • query: The original research query used to evaluate relevance.
    • sourcePreferences (optional): A string describing domains or types of sources the user wants to avoid.
    • model: A LanguageModelV2 instance (from @ai-sdk/provider) used to perform the evaluation.
    • budget (optional): A BudgetState object to track token usage.
    • limiter: A LimitFunction (from p-limit) to control the concurrency of the evaluation requests.

    Returns

    Returns a Promise<string[]> containing the URLs that passed the evaluation.

    const validUrls = await filterSearchResults({
      searchResults: [{ url: 'https://example.com', title: 'Example', description: 'Desc' }],
      query: 'how to build an mcp server',
      sourcePreferences: 'avoid reddit.com',
      model: myAiModel,
      limiter: pLimit(5)
    });