Crawlee

Crawlee is an end-to-end web crawling and scraping library designed to build reliable, fast scrapers that mimic human behavior to avoid bot detection.

Key capabilities include:

• Unified Interface: A single API for both HTTP-based crawling and headless browser-based crawling.
• Resource Management: Automatic scaling based on system resources and a persistent URL queue supporting breadth-first and depth-first traversal.
• Data Handling: Pluggable storage for tabular data and files.
• Stealth & Networking: Integrated proxy rotation, session management, automatic browser-like header generation, and TLS fingerprint replication.
• Extensibility: Customizable lifecycles via hooks and a CLI for project bootstrapping.
• Reliability: Configurable routing, error handling, and retries.
• Deployment: Ready-to-use Dockerfiles.

Crawlee supports both fast HTML parsers (Cheerio, JSDOM) for HTTP crawling and major browser engines (Playwright, Puppeteer) for JavaScript-heavy websites.

Crawlee is a web scraping and browser automation library designed for reliability and speed. Key capabilities include:

HTTP Crawling

• Protocol Support: Zero-config HTTP2 support (including for proxies).
• Fingerprinting: Automatic generation of browser-like headers and replication of browser TLS fingerprints.
• Parsers: Integrated fast HTML parsers like Cheerio and JSDOM.
• API Scraping: Capable of scraping JSON APIs.

Real Browser Crawling

• Rendering: Supports JavaScript rendering and screenshots.
• Modes: Supports both headless and headful (visible) browser modes.
• Fingerprinting: Zero-config generation of human-like fingerprints.
• Abstraction: Use Playwright or Puppeteer with the same unified interface.
• Browsers: Supports Chrome, Firefox, Webkit, and more.

Orchestration & Scaling

• Queue Management: Persistent URL queues supporting breadth-first and depth-first crawling.
• Storage: Pluggable storage for tabular data and files (defaults to ./storage).
• Resource Management: Automatic scaling based on available system resources.
• Stealth: Integrated proxy rotation and session management to avoid bot detection.

@crawlee/http provides a framework for parallel crawling of web pages using plain HTTP requests. It is highly efficient on bandwidth and speed because it does not use a headless browser.

Important Limitation: This crawler downloads pages using plain HTTP requests and does not perform HTML parsing or execute JavaScript. If the target website requires JavaScript to render content, use PuppeteerCrawler or PlaywrightCrawler instead.

Key Concepts

• Input Sources: URLs are provided via Request objects from either a RequestList (static list) or a RequestQueue (dynamic queue for recursive crawling).
• Priority Logic: If both requestList and requestQueue are provided in HttpCrawlerOptions, the crawler first processes the RequestList and automatically enqueues those URLs into the RequestQueue to prevent duplicate crawling.
• Termination: The crawler stops when no more Request objects remain to be processed.
• Concurrency: It uses AutoscaledPool to manage resources. You can configure minConcurrency and maxConcurrency directly in the HttpCrawler constructor.

import { HttpCrawler, Dataset } from '@crawlee/http';

const crawler = new HttpCrawler({
    requestList,
    async requestHandler({ request, response, body, contentType }) {
        // Save the data to dataset.
        await Dataset.pushData({
            url: request.url,
            html: body,
        });
    },
});

await crawler.run([
    'http://www.example.com/page-1',
    'http://www.example.com/page-2',
]);

The @crawlee/linkedom package provides a framework for parallel web crawling using plain HTTP requests and the linkedom DOM implementation.

Key Characteristics

• High Efficiency: Because it uses raw HTTP requests instead of a full browser, it is very fast and uses minimal data bandwidth.
• Non-JS Execution: It does not execute JavaScript. If the target website requires JavaScript to render content, you should use PuppeteerCrawler or PlaywrightCrawler instead.
• Parsing: It downloads HTML, parses it via LinkeDOM, and provides a window object to your requestHandler for data extraction.

Request Management

• Static URLs: You can provide a static list of URLs via requestList.
• Dynamic/Recursive Crawling: You can use a RequestQueue to enable recursive crawling (where the crawler discovers new URLs during execution).
• Priority: If both requestList and requestQueue are provided, the crawler first processes the requestList and automatically enqueues them into the requestQueue to prevent duplicate crawling.

Overview

CheerioCrawler is the fastest and most lightweight crawler in Crawlee. It works by making plain HTTP requests (using got-scraping) and parsing the resulting HTML using Cheerio, which provides a jQuery-like API for Node.js.

When to use CheerioCrawler

Use CheerioCrawler when you need to handle extremely high workloads efficiently and the target website does not require JavaScript to render its content.

Advantages:

• High Performance: Extremely fast and cheap to run (e.g., 500+ pages/minute on 4GB RAM/1 CPU core).
• Low Overhead: Much lower computing requirements compared to browser-based crawlers like PlaywrightCrawler or PuppeteerCrawler.
• Ease of Use: Familiar syntax for anyone who knows jQuery.
• Anti-scraping: Automatically includes features to avoid some anti-scraping bans via the underlying HTTP client.

Disadvantages:

• No JS Rendering: Cannot execute JavaScript. If the content is generated dynamically by JS, use PlaywrightCrawler or PuppeteerCrawler instead.
• No DOM Manipulation: You cannot manipulate the page (e.g., clicking buttons or scrolling) before scraping.
• Risk of Overload: Because it is so fast, it can easily overwhelm target websites if not configured with appropriate concurrency limits.

When building a multi-tenant or multi-proxy API server, you cannot use a single PlaywrightCrawler instance because each instance is tied to a specific proxy configuration. Instead, maintain a Map of crawlers where the key is a stringified version of the crawlerOptions (including proxy settings).

When a new request arrives:

1. Generate a key from the crawlerOptions.
2. Check if a crawler for that key already exists in your map.
3. If it exists, use it; otherwise, initialize a new PlaywrightCrawler with the specific proxyConfiguration and add it to the map.
4. Add the incoming request to that specific crawler's RequestQueue.

const crawlers = new Map<string, PlaywrightCrawler>();

// When a request arrives:
const key = JSON.stringify(crawlerOptions);
const crawler = crawlers.has(key) ? crawlers.get(key)! : await createAndStartCrawler(crawlerOptions);

await crawler.addRequests([request]);