AnyCrawl Documentation

AnyCrawl Webhooks provide real-time HTTP notifications for account events, eliminating the need for polling. When an event occurs, AnyCrawl sends a POST request to your specified endpoint.

Key Features:

• Event Subscription: Subscribe to scraping, crawling, mapping, searching, scheduled tasks, and system events.
• Secure Delivery: Supports HMAC-SHA256 signature verification to ensure authenticity.
• Automatic Retries: Uses an exponential backoff mechanism for failed deliveries.
• Delivery Tracking: Full history of webhook deliveries is available.
• Scope Filtering: Subscribe to all events or specific tasks.
• Custom Headers: Ability to add custom HTTP headers to the webhook request.
• SSRF Protection: Built-in protection against private IP access.

Choosing between scraping and crawling depends on your data requirements:

• /v1/scrape: Used for single URLs. It is synchronous and returns content immediately. Best for single pages to limit costs.
• /v1/crawl: Used for site-wide data. It discovers multiple pages starting from a seed URL and executes asynchronously. Best for large-scale data extraction.

The AnyCrawl API uses consistent naming conventions for requests and responses to ensure predictability.

Parameter Naming

• Request Parameters: Primarily use snake_case (e.g., webhook_url, event_types, cron_expression, max_retries). Note that some specific fields may use existing camelCase names (e.g., timeRange in Search).
• Response Fields: Use snake_case (e.g., task_id, webhook_id, next_execution_at, task_type, cron_expression, is_active).

Always refer to the specific endpoint's parameter table or OpenAPI specification for the authoritative request body schema.

Use the Scheduled Tasks API to automate web scraping, crawling, and search collection using standard cron expressions. This is useful for site monitoring, price tracking, and periodic data aggregation.

Key Features:

• Cron Scheduling: Standard 5-field cron syntax with timezone support (e.g., America/New_York).
• Concurrency Control: Choose between skip (ignore if previous task is running) or queue (wait for previous task to finish).
• Automatic Pause: Tasks pause after consecutive failures to protect resources.
• Webhooks: Real-time notifications via webhook_ids or webhook_url.
• Execution History: Track all runs with status and metrics.

AnyCrawl utilizes two distinct caching layers depending on the API being used:

• Page Cache: Used by the /v1/scrape endpoint and for page-level processing within /v1/crawl.
• Map Cache: Used for URL discovery results generated by the /v1/map endpoint.

The AnyCrawl Crawl API discovers and processes multiple pages starting from a seed URL. It is an asynchronous service: when you initiate a crawl, you receive a job_id immediately. You must then poll the status endpoint and retrieve results using pagination.

Key Features:

• Asynchronous Jobs: Queue a crawl and fetch results later.
• Multiple Engines: Supports auto (default), cheerio, playwright, and puppeteer.
• Flexible Scoping: Control crawling via strategy, max_depth, include_paths, and exclude_paths.
• Per-page Options: Reuse /v1/scrape configurations within scrape_options.
• Pagination: Retrieve results using the skip parameter.

Choosing between the two endpoints depends on your use case:

• /v1/scrape: Used for single URLs. It is synchronous and returns the content immediately.
• /v1/crawl: Used for discovering multiple pages from a seed URL. It is asynchronous and runs as a background job.

AnyCrawl implements two distinct caching layers depending on the API endpoint used:

• Page Cache: Used by /v1/scrape and for page-level processing within /v1/crawl.
• Map Cache: Used by /v1/map specifically for URL discovery results.

AnyCrawl uses two security models for executing template code:

• Untrusted Templates: Executed in an isolated VM sandbox with strict limitations. Most user-created templates fall into this category.
• Trusted Templates: Async functions with controlled access to the browser page. These are only granted to templates verified by AnyCrawl.

AnyCrawl's Scheduled Tasks allow you to automate periodic web scraping, crawling, and searching using standard cron expressions. This is useful for monitoring websites, tracking price changes, or aggregating search results automatically.

Key Features:

• Cron Scheduling: Define execution plans with standard cron syntax.
• Timezone Support: Execute tasks in specific timezones (e.g., Asia/Shanghai).
• Concurrency Control: Choose between skip (skip if previous run is still active) or queue (queue the next run).
• Automatic Pause: Tasks automatically pause after consecutive failures to protect resources.
• Webhook Integration: Receive real-time notifications for task events via webhook_url or webhook_ids.

The AnyCrawl crawl API explores and processes multiple pages starting from a seed URL, converting them into LLM-ready structured data. The process is asynchronous: you receive a job_id immediately, which you then use to poll for status and retrieve results via pagination.

Key Features:

• Asynchronous Tasks: Queue crawl jobs and retrieve results later.
• Multi-Engine Support: Use auto (default), cheerio, playwright, or puppeteer.
• Scope Control: Manage crawling via strategy, max_depth, include_paths, and exclude_paths.
• Single-Page Options: Reuse /v1/scrape settings within the scrape_options field.
• Pagination: Use the skip parameter to stream results and control response size.