mcp-searxng

repository·main·Indexed 21 days ago

https://github.com/ihor-sokoliuk/mcp-searxng

An MCP server that integrates the SearXNG API to provide AI assistants with privacy-respecting web search and URL reading capabilities. It supports web search with pagination and filtering, autocomplete suggestions, and URL content conversion to Markdown (including PDF extraction). Key features include instance failover and fan-out, intelligent in-memory caching, SSRF protection, and optional browser solver support via FlareSolverr or Byparr. It can be deployed via npx, npm, Docker, or as a standalone HTTP server.

Tokens
26.8K
Snippets
62
Records
109
Agent score
76%

What's inside mcp-searxng

  1. Core Features of mcp-searxng

    main

    The mcp-searxng server provides several advanced web capabilities for AI assistants:

    • Web Search: Supports general, news, and article queries with pagination, time-range, language, and safe-search filters. It supports relevance filtering via min_score and configurable output formats via response_format or SEARXNG_DEFAULT_RESPONSE_FORMAT.
    • Instance Failover & Fan-out: Uses SEARXNG_URL to manage replicas. By default, it fails over in order, but can query all healthy replicas in parallel and merge results using the SEARXNG_FANOUT setting.
    • URL Content Reading: Converts URLs to Markdown, including bounded PDF extraction, with support for pagination, section filtering, and heading extraction.
    • Browser Solver Support: Optionally uses FlareSolverr or Byparr to handle complex/protected URLs by acquiring browser sessions and replaying user-agents/cookies.
    • Intelligent Caching: Caches search results and URL content in memory with configurable TTL and LFU (least-frequently-used) eviction.
    • Security & Reliability: Includes SSRF protection (blocking private/internal URLs by default) and supports global or per-tool HTTP/HTTPS proxies.
  2. How mcp-searxng works

    main

    The mcp-searxng project is a standalone Node.js process that acts as an MCP server. It is not a SearXNG plugin. Instead, it sits between your AI assistant and your SearXNG instance(s), communicating with the latter via the HTTP JSON API.

    Architecture Flow:

    1. AI Assistant (e.g., Claude) communicates via the MCP protocol to the mcp-searxng process.
    2. mcp-searxng processes the request and communicates via HTTP JSON API to the configured SearXNG instance(s) (defined by SEARXNG_URL).
  3. Understand the dual-provider failover logic

    main

    The mcp-searxng server supports two browser solvers for bypassing anti-bot protections: FlareSolverr and Byparr.

    When both are configured, the server follows a specific failover chain:

    1. FlareSolverr is always attempted first.
    2. Byparr is only attempted if FlareSolverr returns a 'busy' or 'transient-unavailable' status.

    Failover will NOT occur if:

    • The error is a persistent 4xx status.
    • The request is cancelled.
    • There is a solution-integrity failure.
    • The target returns a non-2xx status after being solved.

    Note that there is no automatic reverse failover (Byparr to FlareSolverr), and provider timeouts and concurrency settings are managed independently for each solver.

  4. Understand the capacity planning boundary for SearXNG

    main

    It is important to distinguish between the MCP adapter and the SearXNG engine. The deployment profiles and measurements provided in this project cover only the MCP adapter process.

    They do not account for:

    • SearXNG engine fan-out
    • Redis usage
    • Result rendering
    • Bot detection
    • Upstream network behavior

    To size or manage the SearXNG search service itself, refer to the SearXNG installation documentation. For integration boundaries, see the self-hosted-searxng.md guide.

  5. Use HTML fallback for JSON-disabled instances

    main

    If SEARXNG_HTML_FALLBACK is set to true, mcp-searxng will attempt to retry a search by parsing the standard HTML results page if the JSON request fails with:

    • HTTP 403 (JSON disabled or access control)
    • HTTP 404 (Endpoint mismatch)
    • A successful response that is not valid JSON

    Note: The fallback provides title, URL, and snippet data, but marks the output with sourceFormat: "html" and omits metadata like relevance scores or engine names. It does not bypass authentication, network errors, 429 (rate limits), or 5xx errors.

  6. Understand URL Reader security and limitations

    main

    The web_url_read tool includes several security and operational constraints:

    Security

    • Private URL Blocking: By default, the tool blocks requests to localhost, loopback, private IPv4 ranges (e.g., 10.0.0.0/8), CGNAT, and other special-purpose/internal addresses. This includes checking redirects.
    • DNS Rebinding Protection: For direct requests, DNS answers are validated before connecting, and the connection is pinned to the validated answer.
    • Enabling Private Access: Set MCP_HTTP_ALLOW_PRIVATE_URLS=true only if internal URL reads are intentional.

    Resource Limits

    • Content Length: URL_READ_MAX_CONTENT_LENGTH_BYTES limits the response body size (measured after decompression).
    • PDF Extraction:
      • Limited to the lower of URL_READ_MAX_CONTENT_LENGTH_BYTES or 16 MiB.
      • Extraction is limited to 500 pages.
      • No OCR is performed.
      • Concurrency: At most two PDF extractions run concurrently per process. Additional requests return: PDF text extraction is busy; try again later.
      • Timeouts: PDF parsing has a 30-second worker budget after the response body is complete.
  7. Choose an MCP connection mode for mcp-searxng

    main

    You can connect supported MCP clients to a SearXNG endpoint using one of three modes:

    1. NPX/STDIO: The client starts the npm package locally. Requires Node.js 20 or newer.
    2. Docker/STDIO: The client starts the published container locally. Requires Docker.
    3. HTTP: The client connects to an independently running Streamable HTTP server. The server operator (not the client) sets the SEARXNG_URL.

    Note on Security: For clients like Claude Desktop and Cursor, this guide recommends using local STDIO instead of remote HTTP. This is because mcp-searxng uses static bearer-token hardening rather than MCP OAuth, and STDIO avoids exposing the server to the network.

  8. Observe and adjust MCP deployment performance

    main

    After deploying the MCP server using a profile, you should sample the container's resource usage during representative traffic to ensure it is sized correctly.

    For Docker run deployments

    Use docker stats to check resource usage and docker inspect to check for Out-Of-Memory (OOM) kills or restarts:

    docker stats --no-stream mcp-searxng-profile
    docker inspect mcp-searxng-profile --format '{{.State.OOMKilled}} {{.RestartCount}}'

    For Docker Compose HTTP deployments

    When using multiple Compose files, address the service through Compose rather than assuming a container name. Use the following commands to check stats and inspect the specific container ID:

    docker compose \
      -f docker-compose.yml \
      -f docker-compose.http.yml \
      -f docker-compose.resources.yml \
      stats --no-stream mcp-searxng
    
    container_id="$(docker compose \
      -f docker-compose.yml \
      -f docker-compose.http.yml \
      -f docker-compose.resources.yml \
      ps -q mcp-searxng)"
    
    docker inspect "$container_id" --format '{{.State.OOMKilled}} {{.RestartCount}}'

    Tuning Guidelines

    • Increase Memory: If normal peaks approach the limit or the container is OOMKilled.
    • Increase CPU: If latency rises due to sustained CPU throttling.
    • Reduce Cache/Limits: If retained content (cache caps or page/result limits) is causing memory pressure.
    • Scale Out: Deploy separate MCP instances if clients require different security policies, SearXNG endpoints, cache lifecycles, or if a single process cannot meet concurrency targets.
  9. Configure Cline HTTP

    main

    Cline supports Streamable HTTP via type: "streamableHttp". Replace the placeholder token with your actual token and protect the configuration file. An empty autoApprove list ensures tool calls require manual approval.

    {
      "mcpServers": {
        "searxng-http": {
          "type": "streamableHttp",
          "url": "https://mcp.example.com/mcp",
          "headers": {
            "Authorization": "Bearer REPLACE_WITH_TOKEN"
          },
          "disabled": false,
          "autoApprove": []
        }
      }
    }
  10. Configure TLS/Corporate CA for Node.js

    main

    If you are behind a TLS-inspecting corporate proxy (e.g., Zscaler, Netskope) and encounter UNABLE_TO_GET_ISSUER_CERT_LOCALLY errors, you must provide the proxy's root CA to the Node.js runtime.

    On Linux/macOS

    mcp-searxng automatically detects system CA bundles from standard paths (e.g., /etc/ssl/certs/ca-certificates.crt).

    On Windows

    1. Export the proxy root CA from the Windows certificate store to a .pem file.
    2. Set the NODE_EXTRA_CA_CERTS environment variable to the path of that PEM file.

    Warning: Never set NODE_TLS_REJECT_UNAUTHORIZED=0 as it disables all TLS validation and makes connections vulnerable to interception.

    # Export from Windows cert store to PEM:
    $cert = Get-ChildItem Cert:\LocalMachine\Root | Where-Object { $_.Subject -match "YourCorp" } | Select-Object -First 1
    [System.IO.File]::WriteAllBytes("$env:USERPROFILE\corp-ca.cer", $cert.RawData)
    certutil -encode "$env:USERPROFILE\corp-ca.cer" "$env:USERPROFILE\corp-ca.pem"
    {
      "env": {
        "SEARXNG_URL": "https://searxng.example.com",
        "NODE_EXTRA_CA_CERTS": "C:\\Users\\you\\corp-ca.pem"
      }
    }
  11. Configure the SearXNG MCP Server with a combined configuration

    main

    To configure the SearXNG MCP server, provide an environment variable object within your MCP client configuration. The configuration includes several independent groups: Core settings, Authentication, Timeouts, Tool Schema, Search Defaults, Search Result Controls, Search Compatibility, URL Reader Controls, and User-Agent settings.

    Important Dependencies:

    • If you enable security hardening with MCP_HTTP_HARDEN=true, you must also configure MCP_HTTP_AUTH_TOKEN and MCP_HTTP_ALLOWED_ORIGINS together.
    • Network/Proxy settings like MCP_HTTP_ALLOWED_HOSTS and MCP_HTTP_TRUST_PROXY depend on your specific network topology. For example, MCP_HTTP_TRUST_PROXY=1 assumes exactly one trusted proxy hop. Without this, clients might be able to spoof X-Forwarded-For headers, affecting IP-based rate limiting.
    {
      "mcpServers": {
        "searxng": {
          "command": "npx",
          "args": ["-y", "mcp-searxng"],
          "env": {
            "SEARXNG_URL": "https://searxng.example.com",
            "AUTH_USERNAME": "legacy-fallback-user",
            "AUTH_PASSWORD": "legacy-fallback-password",
            "SEARXNG_FANOUT": "false",
            "SEARXNG_TIMEOUT_MS": "10000",
            "FETCH_TIMEOUT_MS": "10000",
            "SEARXNG_LITE_TOOLS": "false",
            "SEARXNG_DEFAULT_LANGUAGE": "en",
            "SEARXNG_DEFAULT_SAFESEARCH": "0",
            "SEARXNG_DEFAULT_RESPONSE_FORMAT": "text",
            "SEARXNG_MAX_RESULTS": "10",
            "SEARXNG_MAX_RESULT_CHARS": "500",
            "SEARCH_CACHE_TTL_MS": "86400000",
            "SEARCH_CACHE_MAX_ENTRIES": "200",
            "SEARXNG_HTML_FALLBACK": "false",
            "URL_READ_MAX_CHARS": "2000",
            "URL_READ_MAX_CONTENT_LENGTH_BYTES": "5242880",
            "FLARESOLVERR_URL": "http://flaresolverr:8191",
            "FLARESOLVERR_TIMEOUT_MS": "60000",
            "FLARESOLVERR_MAX_CONCURRENT_REQUESTS": "2",
            "CACHE_TTL_MS": "86400000",
            "CACHE_MAX_ENTRIES": "500",
            "USER_AGENT": "MyBot/1.0",
            "SEARCH_USER_AGENT": "MySearchBot/1.0",
            "URL_READER_USER_AGENT": "Mozilla/5.0 (compatible; MyBot/1.0)",
            "SEARCH_HTTP_PROXY": "http://search-proxy.company.com:8080",
            "SEARCH_HTTPS_PROXY": "http://search-proxy.company.com:8080",
            "URL_READER_HTTP_PROXY": "http://reader-proxy.company.com:8080",
            "URL_READER_HTTPS_PROXY": "http://reader-proxy.company.com:8080",
            "HTTP_PROXY": "http://global-proxy.company.com:8080",
            "HTTPS_PROXY": "http://global-proxy.company.com:8080",
            "NO_PROXY": "localhost,127.0.0.1,.local,.internal",
            "MCP_HTTP_PORT": "3000",
            "MCP_HTTP_HOST": "0.0.0.0",
            "MCP_HTTP_TRUST_PROXY": "1",
            "MCP_HTTP_STATELESS": "false",
            "MCP_HTTP_STATELESS_MAX_IN_FLIGHT": "16",
            "MCP_HTTP_STATELESS_MAX_IN_FLIGHT_PER_IP": "8",
            "MCP_HTTP_STATELESS_REQUEST_TIMEOUT_MS": "900000",
            "MCP_RATE_WINDOW_MS": "60000",
            "MCP_RATE_INIT_MAX": "20",
            "MCP_RATE_SESSION_MAX": "300",
            "MCP_HTTP_HARDEN": "true",
            "MCP_HTTP_AUTH_TOKEN": "replace-me",
            "MCP_HTTP_ALLOWED_ORIGINS": "https://app.example.com",
            "MCP_HTTP_ALLOWED_HOSTS": "app.example.com",
            "MCP_HTTP_ALLOW_PRIVATE_URLS": "false",
            "MCP_HTTP_EXPOSE_FULL_CONFIG": "false"
          }
        }
      }
    }
  12. Select an MCP Deployment Profile

    main

    The mcp-searxng process can be sized using three starting profiles based on your expected workload. These profiles are starting points for the MCP process only and do not include sizing for SearXNG, Redis, or proxies.

    Note: If you use PDF extraction or browser-solver features, you should increase the memory limits beyond these baseline recommendations to account for additional overhead.

    | Profile | Intended MCP workload | CPU starting range | Memory starting range |
    | --- | --- | ---: | ---: |
    | Small | 1-2 mostly sequential clients, one SearXNG URL, modest caches | 0.25-0.50 CPU | 192-256 MiB |
    | Balanced | About 4 concurrent clients, replicas in failover mode, default-sized caches | 0.50-1.00 CPU | 256-384 MiB |
    | Research-heavy | About 8 concurrent sessions, fan-out or large caches, frequent 12 KiB page reads | 1.00-2.00 CPU | 512-768 MiB |