Crawlee for Python

Crawlee is an end-to-end web crawling and scraping library designed to build reliable, fast scrapers. It provides tools to crawl links, scrape data, and store it in machine-readable formats while managing technical complexities like bot detection.

Key capabilities include:

• Unified Interface: Supports both HTTP and headless browser crawling.
• Automatic Parallelism: Scales crawling based on available system resources.
• Resilience: Automatic retries on errors or blocks, integrated proxy rotation, and session management.
• Request Management: Configurable request routing and a persistent URL queue.
• Storage: Pluggable storage for both tabular data and files.
• Developer Experience: Written in Python with full type hint coverage for IDE autocompletion and static type checking, built on standard asyncio.

HTTP crawlers in Crawlee are designed for extracting data from server-rendered websites that do not require JavaScript execution. They fetch HTML content via HTTP clients and use various parsing libraries to extract data.

All HTTP crawlers inherit from the AbstractHttpCrawler base class. Key implementations include:

• BeautifulSoupCrawler: For fault-tolerant parsing of malformed HTML.
• ParselCrawler: For high-performance XPath and CSS selector support.
• HttpCrawler: For raw HTTP responses (JSON, XML, binary) without automatic parsing.
• FileDownloadCrawler: For downloading files.

If you need to execute JavaScript for client-side rendered content, use the Playwright crawler instead.

Crawlee provides several specialized HTTP crawler types depending on your parsing needs:

• BeautifulSoupCrawler: Best for fault-tolerant HTML parsing using the BeautifulSoup library.
• ParselCrawler: Optimized for high-performance extraction using XPath and CSS selectors.
• HttpCrawler: Designed for processing raw responses when specialized parsing is required.
• AbstractHttpCrawler: A base class used to build fully custom crawlers for specialized parsing requirements.

Crawlee uses two main layers for data management:

1. Storages: High-level interfaces for interacting with data.
   • RequestQueue: Manages URLs and deduplication.
   • Dataset: Stores structured, append-only scraping results (like table rows).
   • KeyValueStore: Stores key-value pairs.
2. Storage clients: Backend implementations (e.g., MemoryStorageClient, FileSystemStorageClient) that handle actual persistence.

Storages come in two flavors:

• Named storages: Persistent across runs; useful for sharing data between different crawler executions.
• Unnamed storages: Scoped to a single run; automatically purged at the start of each run by default (purge_on_start enabled).

Crawlee provides several storage client implementations to manage datasets, key-value stores, and request queues:

• FileSystemStorageClient: Persistent file system storage with in-memory caching (Default).
• MemoryStorageClient: In-memory storage with no persistence (ideal for testing).
• SqlStorageClient: Persistent storage using SQL databases (SQLite, PostgreSQL, MySQL, MariaDB). Requires extra dependencies.
• RedisStorageClient: Persistent storage using Redis v8.0+. Requires extra dependencies.
• ApifyStorageClient: Manages storage on the Apify platform (via Apify SDK).

The request_loaders sub-package extends RequestQueue to provide advanced tools for managing URLs and requests. Request loaders define how requests are fetched and stored, allowing you to read URLs from files, external APIs, or multiple sources.

Key abstract classes include:

• RequestLoader: The base interface for reading requests.
• RequestManager: Extends RequestLoader with write capabilities (adding/reclaiming requests).
• RequestManagerTandem: Combines a read-only RequestLoader with a writable RequestManager.

Common implementations:

• RequestList: A lightweight implementation for managing static lists of URLs.
• SitemapRequestLoader: A specialized loader for XML and plain-text sitemaps following the Sitemaps protocol.

Use PlaywrightCrawler when you need a real browser to render pages. This is essential for:

• Dynamic content rendering: Pages relying heavily on JavaScript.
• Anti-scraping protection: Sites using JavaScript-based security.
• Complex cookie management: Sites with specific session or cookie requirements.

Unlike ParselCrawler or BeautifulSoupCrawler, which are HTTP-based, PlaywrightCrawler is built on top of the Playwright library to handle client-side rendered sites.

The ServiceLocator acts as a central registry for global services in Crawlee, ensuring consistent configuration across all components. It manages three core services that are lazily initialized with defaults upon first access:

• Configuration: Provides access to application-wide settings (e.g., timeouts, logging levels, persistence intervals). Can be configured via code or environment variables.
• StorageClient: The backend implementation for all storages. It provides a unified interface for Dataset, KeyValueStore, and RequestQueue regardless of the underlying storage type.
• EventManager: Coordinates internal framework events. It allows you to register listeners for events like aborting, migrating, system info, or browser events like page created and page closed.