DOCSEARCHbyAlgolia
Submit Docs

Spider API Reference

website·Indexed 34 days ago

https://spider.cloud/docs/api

Technical reference for the Spider API, covering authentication, base URL configurations, and the POST /crawl endpoint. Includes details on target traversal, fetch modes, CSS extraction, network identity settings, proxy configurations, and standardized error codes for AI and unlimited tiers.

Tokens
3.2K
Snippets
17
Records
22
Agent score
21%

What's inside Spider

  1. Use the Unlimited plan routes

    The Unlimited plan uses flat-rate concurrency seats instead of per-request credits. Use the /unlimited/ prefix (e.g., /unlimited/scrape, /unlimited/crawl, /unlimited/links).

    Key characteristics:

    • No queueing: If all seats are busy, the API returns 429 with concurrency_limit_reached.
    • Monitoring: Responses include X-Concurrency-Limit and X-Concurrency-Active headers.
    • Constraints: AI parameters (prompt, extraction_schema) are not supported on these routes; use /ai/* for AI work.

  2. Use AI-driven prompt extraction

    Prompt-based extraction is available via the /ai/ routes (e.g., /ai/scrape, /ai/crawl). These routes require an active AI plan. To use them, pass a prompt and optionally an extraction_schema along with the standard parameters of the base route. Without a subscription, these return a 402 error with code ai_subscription_required.

  3. Use the keyless scrape tier

    The POST /scrape endpoint can be used without an account or Authorization header. This tier is rate-limited and only supports the /scrape route. Any other route will return a 402 error. If a 429 error occurs, the Retry-After header should be honored.
    curl -X POST https://api.spider.cloud/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "return_format": "markdown"}'

  4. Configure output formats and content extraction

    Define the format and cleanliness of the returned data. Note that return_format defaults to raw (unmodified HTML). For LLM usage, explicitly set it to markdown.
    {
  "return_format": "markdown", // Options: "raw", "markdown", "commonmark", "text", "html2text", "xml", "bytes", "screenshot", "empty"
  "readability": true,           // Strip boilerplate
  "metadata": true,             // Include title, description, keywords, og:image
  "return_page_links": true,    // Include links found on page
  "root_selector": "#main",   // Limit extraction to this subtree
  "exclude_selector": ".ads", // Drop matching content
  "filter_output_main_only": true, // Drop nav, aside, footer
  "clean_html": true            // Strip unwanted attributes
}

  5. Extract specific data using css_extraction_map

    The css_extraction_map allows you to define named selector groups keyed by URL-path patterns. The pattern "/" matches all URLs. If multiple selectors are provided for a name, they act as fallbacks. Extracted data is returned in the css_extracted field of the response.
    {
  "url": "https://store.example.com/product/123",
  "request": "browser",
  "css_extraction_map": {
    "/": [
      { "name": "title",  "selectors": ["h1.product-title"] },
      { "name": "price",  "selectors": [".price-value"] },
      { "name": "images", "selectors": ["img.hero-image", "img.gallery"] }
    ]
  }
}

  6. Interpret Spider API HTTP response codes

    The Spider API uses standard HTTP status codes to indicate the result of a request. Note that attempts are billed based on bytes and compute consumed, regardless of whether the response indicates a success or an error.
    CodeMeaning
    200Success.
    204No content.
    400Bad parameters or malformed JSON. On /unlimited/*, AI params were sent.
    401Missing, invalid, or expired key.
    402Out of credits, or a keyless request to any route other than /scrape. Also /ai/* without an active AI plan (ai_subscription_required).
    403/unlimited/* without an active Unlimited plan (unlimited_plan_required / unlimited_plan_inactive), or a funded key refused for any other reason.
    413Payload too large.
    429Rate limited. On /unlimited/*, all concurrency seats are busy.
    500Server error.
    503Temporarily unavailable.

  7. Spider API Route Map

    The Spider API provides several POST endpoints for different web extraction tasks. Most paths also accept an optional /v1/ prefix (e.g., /v1/crawl is equivalent to /crawl).
    POST /scrape       - Single page extraction (works keyless)
POST /crawl        - Multiple pages from one origin
POST /links        - Extract URLs only
POST /search       - Web search with optional fetching
POST /screenshot   - Page image (base64 or binary)
POST /unblocker    - Bypass bot walls (higher credit cost)
POST /transform    - Process provided HTML (no fetching)
POST /fetch/{dom}/{path} - Fetch page using stored domain config
GET /data/credits  - Check remaining credit balance

  8. Extract links from a page using POST /links

    The /links endpoint uses the same parameters as /crawl but returns a list of discovered URLs instead of the page content.
    [
  {
    "url": "https://example.com",
    "status": 200,
    "error": null,
    "links": ["https://example.com/about", "https://example.com/pricing"],
    "costs": { "total_cost": 0.00011 }
  }
]

  9. Configure network identity and proxies for Spider API

    The Spider API provides several parameters to control the network identity and proxy settings of a request. Premium proxies increase cost by 1.5x. Residential proxies cost 1.2x, mobile 2x, and ISP/datacenter 1.2x. Using a remote proxy can save 50% on data transfer.
    {
  "proxy_enabled": true, // Enable premium proxies
  "proxy": "residential", // Options: residential, mobile, isp, datacenter
  "remote_proxy": "http://user:pass@host:port", // Use your own proxy
  "country_code": "gb", // ISO code for egress
  "user_agent": "Custom User Agent String",
  "cookies": "session_id=12345",
  "session": true, // Persist headers/cookies across crawl (default: true)
  "preserve_host": false, // Keep original HOST header
  "request_timeout": 5000, // Per-request timeout in ms
  "timeout": 30000, // Overall timeout in ms
  "request_max_retries": 3,
  "request_redirect_limit": 5,
  "cache": true // Default true (2-day window). Set to false for fresh fetch.
}

  10. Understand Spider API rate limits and quotas

    Rate limits vary by plan. All responses include RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset (seconds until reset) headers. For 429 errors, honor the Retry-After header.

    • Pay as you go: 10,000 requests per minute per account.
    • Enterprise: 50,000 requests per minute (higher available on request).
    • Unlimited: Limited by concurrency (requests in flight) rather than requests per minute. Returns 429 when all concurrency seats are busy. Usage is reported via X-Concurrency-Limit and X-Concurrency-Active headers.
    • AI Studio (/ai/*): Per-second caps based on tier: Starter (1 req/s), Lite (5 req/s), Standard (10 req/s), Scale (25 req/s), Enterprise (custom > 25 req/s).