Fetch MCP Server

repository·main·Indexed 21 days ago

https://github.com/zcaceres/fetch-mcp

An MCP (Model Context Protocol) server and CLI for fetching web content in HTML, Markdown, JSON, and plain text formats, as well as YouTube transcripts. It includes a 'readable' mode using Mozilla Readability to extract main article content while stripping boilerplate. The tool provides security protections against SSRF and allows configuration of response limits and proxies via environment variables or request parameters.

Tokens
4.2K
Snippets
19
Records
24
Agent score
72%

What's inside mcp-fetch-server

  1. Install Fetch MCP Server as a CLI

    main

    You can run the fetcher directly from your terminal using npx or by installing the package globally via npm.

    Using npx:

    npx mcp-fetch <command> <url> [flags]

    Global installation:

    npm install -g mcp-fetch-server
    mcp-fetch <command> <url> [flags]
  2. Install Fetch MCP Server as an MCP server

    main

    To use Fetch MCP Server with an MCP client (like Claude Desktop), add the following configuration to your mcpServers settings. This uses npx to run the mcp-fetch-server package directly.

    {
      "mcpServers": {
        "fetch": {
          "command": "npx",
          "args": ["mcp-fetch-server"]
        }
      }
    }
  3. Security restrictions in Fetcher

    main

    The Fetcher class implements several security checks to prevent SSRF (Server-Side Request Forgery):

    1. Protocol Validation: Only http: and https: protocols are allowed.
    2. Private Address Blocking: Requests to localhost or private IP ranges (e.g., 192.168.x.x) are blocked.
    3. DNS Rebinding Protection: The hostname is resolved via DNS, and if the resulting IP is a private address, the request is blocked.
    4. Response Size Limits: Requests are aborted if the content-length header exceeds maxResponseBytes or if the body exceeds the limit during streaming.
  4. Configure RequestPayload limits and offsets

    main

    When using Fetcher methods, you can control the output size and position using these keys in the RequestPayload:

    • max_length: The maximum number of characters to return. Defaults to downloadLimit.
    • start_index: The character index to start from. Defaults to 0.

    These limits are applied to the final processed string (HTML, Markdown, Text, etc.) before returning.

  5. Configure Fetch MCP Server Environment Variables

    main

    You can control the server's default behavior using environment variables in your MCP client configuration.

    VariableDescription
    DEFAULT_LIMITDefault character limit for responses (default: 5000, set to 0 for no limit)
    MAX_RESPONSE_BYTESMaximum response body size in bytes (default: 10485760 / 10 MB)

    Example configuration with a custom limit:

    {
      "mcpServers": {
        "fetch": {
          "command": "npx",
          "args": ["mcp-fetch-server"],
          "env": {
            "DEFAULT_LIMIT": "50000"
          }
        }
      }
    }
  6. CLI Usage Examples

    main

    Common patterns for using the mcp-fetch CLI:

    Fetch a page as markdown:

    mcp-fetch markdown https://example.com

    Extract article content without boilerplate:

    mcp-fetch readable https://example.com/blog/post

    Get a YouTube transcript in Spanish:

    mcp-fetch youtube https://www.youtube.com/watch?v=dQw4w9WgXcQ --lang es

    Fetch with a length limit:

    mcp-fetch html https://example.com --max-length 10000

    Fetch through a proxy:

    mcp-fetch json https://api.example.com/data --proxy http://proxy:8080
  7. Configure fetch limits via environment variables

    main

    You can control the default behavior of the server by setting the following environment variables:

    • DEFAULT_LIMIT: Sets the default max_length for requests. Defaults to 5000 if not set or invalid.
    • MAX_RESPONSE_BYTES: Sets the maximum allowed response size in bytes. Defaults to 10485760 (10MB) if not set or invalid.
  8. Reference: Fetch MCP Server Tools and Parameters

    main

    All tools in the Fetch MCP Server accept the following common parameters:

    ParameterTypeRequiredDescription
    urlstringYesURL to fetch
    headersobjectNoCustom headers to include in the request
    max_lengthnumberNoMaximum characters to return (default: 5000)
    start_indexnumberNoStart from this character index (default: 0)
    proxystringNoProxy URL (e.g. http://proxy:8080)

    Available Tools:

    • fetch_html: Returns raw HTML content.
    • fetch_markdown: Returns content converted to Markdown.
    • fetch_txt: Returns plain text (removes HTML tags, scripts, and styles).
    • fetch_json: Returns the JSON response from a URL.
    • fetch_readable: Uses Mozilla Readability to extract main article content as Markdown, stripping navigation, ads, and boilerplate.
    • fetch_youtube_transcript: Fetches YouTube captions/transcripts. Accepts an additional lang parameter (default: "en").
  9. Reference: Fetch MCP Server CLI Commands and Flags

    main

    When using the CLI, use the following commands and flags:

    Commands:

    • html: Fetch raw HTML
    • markdown: Fetch Markdown
    • readable: Fetch article content via Readability
    • txt: Fetch plain text
    • json: Fetch JSON
    • youtube: Fetch YouTube transcript

    Flags:

    • --max-length <N>: Maximum characters to return
    • --start-index <N>: Start from this character index
    • --proxy <URL>: Proxy URL
    • --lang <code>: Language code for YouTube transcripts (default: en)
    • --help: Show help message
    • --version: Show version
  10. Use fetch_readable for clean article content

    main

    When you need to extract the core content of a blog post or news article without the surrounding noise (ads, sidebars, navigation), use the fetch_readable tool. It uses Mozilla Readability to parse the page and returns the result as Markdown.

    {
      "name": "fetch_readable",
      "arguments": {
        "url": "https://example.com/blog-post",
        "max_length": 5000
      }
    }
  11. Parse CLI arguments with parseArgs()

    main

    If you are integrating the CLI logic into your own TypeScript application, you can use the parseArgs function to transform process.argv into a structured ParsedArgs object.

    ParsedArgs Interface:

    • subcommand: One of html, markdown, readable, txt, json, or youtube.
    • url: The target URL string.
    • maxLength?: (Optional) Number of characters to return.
    • startIndex?: (Optional) Starting character index.
    • proxy?: (Optional) Proxy URL string.
    • lang?: (Optional) Language code string.
    import { parseArgs, type ParsedArgs } from './cli.js';
    
    const args: ParsedArgs = parseArgs(process.argv.slice(2));
    console.log(args.subcommand, args.url);
  12. Configure YouTubeTranscriptPayload for the YouTube transcript tool

    main

    When requesting a YouTube transcript, use the YouTubeTranscriptPayload schema. It extends the standard RequestPayload with language support.

    Fields:

    • url (string, required): The URL of the YouTube video.
    • headers (Record<string, string>, optional): Custom HTTP headers.
    • max_length (number, optional): Maximum characters to download. Defaults to downloadLimit.
    • start_index (number, optional): Starting index. Defaults to 0.
    • proxy (string, optional): Proxy URL.
    • lang (string, optional): The language code for the transcript (e.g., 'en', 'es'). Defaults to 'en'.
    const payload: YouTubeTranscriptPayload = {
      url: 'https://www.youtube.com/watch?v=example',
      lang: 'es',
      max_length: 5000
    };