Spider - High-Performance Web Crawling and Scraping Engine

To route all crawling through Spider Cloud (providing managed proxies, anti-bot bypass, and CAPTCHA handling), set the SPIDER_API_KEY environment variable in your MCP configuration.

{
  "mcpServers": {
    "spider": {
      "command": "spider-mcp",
      "env": {
        "SPIDER_API_KEY": "your-api-key"
      }
    }
  }
}

Depending on your environment, you can install Spider using the following commands:

• Rust library: cargo add spider
• Command-line tool: cargo install spider_cli
• Node.js package: npm i @spider-rs/spider-rs
• Python package: pip install spider_rs
• MCP server (Claude, Cursor, etc.): cargo install spider_mcp

To use the spider_agent skills integration (which includes web challenges and pre-evaluate JS) within the main spider crate, you must enable the corresponding feature flags in your Cargo.toml.

Use agent_skills for standard skill support, or agent_skills_s3 if you need to load skills from S3-compatible storage.

To use Spider in your Rust application, add it to your Cargo.toml. For cloud-based crawling features, include the spider_cloud feature.

[dependencies]
spider = { version = "2", features = ["spider_cloud"] }

You can install the spider_mcp Model Context Protocol server via Cargo or build it from source.

To install the default version:

cargo install spider_mcp

To build from source:

cargo build -p spider_mcp --release

For a minimal build that uses HTTP only and excludes Chrome rendering, use:

cargo install spider_mcp --no-default-features

To use spider_mcp with Claude Code, add the server configuration to your ~/.claude/settings.json file.

{
  "mcpServers": {
    "spider": {
      "command": "spider-mcp",
      "args": [],
      "env": {}
    }
  }
}

You can explicitly control how Spider renders pages:

• HTTP mode (Default): Fast, non-rendering mode.
• Browser mode: Uses headless Chrome for rendering (requires chrome feature).
• Force HTTP-only: Forces HTTP mode even in builds that include Chrome support.

Examples:

Run in default HTTP mode:

spider --url https://choosealicense.com crawl --output-links

Run in Browser mode (headless):

spider --url https://choosealicense.com --headless crawl --output-links

Force HTTP-only mode:

spider --url https://choosealicense.com --http crawl --output-links

Add spider_agent to your Cargo.toml dependencies. You must enable specific features for the providers you intend to use (e.g., openai, search_serper).

[dependencies]
spider_agent = { version = "0.1", features = ["openai", "search_serper"] }

You can install the Spider CLI binary using cargo. There are two main installation options:

1. Default install: Includes Chrome support for browser rendering.
2. Smart mode: Uses HTTP first and falls back to a browser if necessary.

Dependencies (Linux/Ubuntu): You must have pkg-config and OpenSSL (1.0.1, 1.0.2, 1.1.0, or 1.1.1) installed.

# Install pkg-config on Ubuntu
apt install pkg-config

# Default install (includes chrome support)
cargo install spider_cli

# Optional smart mode (HTTP first, browser fallback)
cargo install -F smart spider_cli

# Default install (includes chrome support)
cargo install spider_cli

# optional smart mode (HTTP first, browser fallback)
cargo install -F smart spider_cli

Add spider_agent_html to your Cargo.toml dependencies to use streaming HTML processing utilities.

[dependencies]
spider_agent_html = "0.1"

Using a custom global allocator can improve throughput by 30-60% under high concurrency.

• For mixed crawl+parse workloads, use mimalloc.
• For pure crawl workloads, use tikv-jemallocator.

Example using mimalloc:

[dependencies]
mimalloc = { version = "0.1", default-features = false }

#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;

#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
// or: static GLOBAL: tikv_jemallocator::Jemalloc = tikv_jemallocator::Jemalloc;

Optimize the cleaning process for a specific downstream task using clean_html_with_profile_and_intent. This allows you to combine a profile with a CleaningIntent (e.g., Extraction) to tailor the output.

use spider_agent_html::clean_html_with_profile_and_intent;
use spider_agent_types::{HtmlCleaningProfile, CleaningIntent};

// More aggressive cleaning for extraction tasks
let result = clean_html_with_profile_and_intent(
    html,
    HtmlCleaningProfile::Default,
    CleaningIntent::Extraction,
);