Frontera Documentation

Frontera is a crawl frontier implementation designed for accumulating URLs and links before they are downloaded from the web. It is a pure Python 3 implementation built for online processing and distributed architectures.

Key architectural features include:

• Distributed Architecture: Supports distributed spiders and backends.
• Customizable Crawling Policy: Allows for precise tuning of crawling logic.
• Extensive Backend Support: Supports relational databases (MySQL, PostgreSQL, SQLite, etc.) via SQLAlchemy, as well as HBase key-value storage.
• Message Bus Implementations: Uses ZeroMQ and Kafka for communication between distributed crawlers.
• Scrapy Integration: Designed for easy integration with the Scrapy framework.
• Graph Management: Supports crawling emulation using fake sitemaps via the Graph Manager to tune crawling logic.

Frontera is a web crawling framework designed for large-scale online web crawling. It manages the crawl frontier (the collection of URLs to be visited) and provides distribution and scaling primitives.

Key responsibilities include:

• Logic and Policies: Managing the rules of the crawl.
• Link Management: Storing and prioritizing links extracted by crawlers to determine the next pages to visit.
• Distributed Execution: Capable of operating in both single-process and distributed modes.

The Graph Manager is a tool used to represent web sitemaps as a directed graph, where nodes are pages and edges are links. It is primarily used to test frontiers by "faking" crawler requests and responses without an actual crawler. You can define sites manually or use the Scrapy Recorder to reproduce previous crawls.

To use it, instantiate graphs.Manager() and use add_site to populate it with a site structure.

A Canonical URL Solver is a specialized middleware in Frontera responsible for identifying the canonical URL of a document. It modifies request or response metadata to ensure that different URLs pointing to the same document (e.g., via different redirect chains or multiple access paths) are treated as a single entity.

Key characteristics:

• Execution Order: It always executes last in the middleware chain, immediately before calling Backend methods.
• Purpose: It prevents metadata record duplication and prevents confusing crawler behavior caused by multiple URLs representing the same content.
• Configuration: It is instantiated during Frontera Manager initialization using the class specified in the CANONICAL_SOLVER setting.

Frontera is suitable for scenarios where you need to decouple URL management from the spider logic. Use cases include:

• URL Ordering/Queueing Isolation: When you need to manage ordering or queueing remotely or across a distributed cluster of spiders.
• URL (meta)data Storage: When you need to persist URL metadata to enable pausing and resuming crawls.
• Advanced Ordering Logic: When complex URL prioritization logic is too difficult to maintain directly within the spider or fetcher code.

A DistributedBackend (inheriting from frontera.core.components.DistributedBackend) acts as an abstraction layer that separates high-level crawling strategies from low-level storage APIs.

While multiple Middleware instances can be active, only one DistributedBackend can be used per frontier. The DistributedBackend manages and holds references to four inner components that handle specific data types:

1. Queue: A priority queue for persisting scheduled requests.
2. Metadata: Stores the contents of the crawl.
3. States: Stores link states (represented as short integers).
4. DomainMetadata: Stores per-domain information like flags, counters, or robots.txt content.

The FrontierManager communicates with the active backend using hooks for Request and Response processing after the middleware layer.

The FrontierTester is a helper class designed to facilitate easy testing of a Frontier by running a simulated (fake) crawl. It uses a Graph Manager instance to provide fake crawl data, simulating page responses and link discovery.

When the run() method is called, the tester performs the following lifecycle:

1. Adds all seed URLs from the provided graph to the frontier.
2. Requests the next set of pages from the frontier.
3. Fakes a page response and informs the frontier about the crawl results and discovered links.

This cycle repeats until either the crawl finishes or the frontier is exhausted. After execution, the sequence of crawled pages is stored in the sequence attribute as a list of Request objects.

>>> tester = FrontierTester(frontier, graph)
>>> tester.run()
>>> print(tester.sequence)  # Returns a list of Request objects

Frontera provides several architectural features for building scalable crawlers:

Architecture & Scaling

• Pluggable Backend: Separates low-level backend access logic from the crawling strategy.
• Run Modes: Supports both single-process and distributed execution.
• Message Bus Abstraction: Allows for custom transports, with built-in support for Apache Kafka and ZeroMQ.
• Online Operation: Processes small request batches with parsing performed immediately after fetching.

Built-in Components

• Backends: Supports SqlAlchemy, Redis, and HBase.
• Crawling Strategies: Includes breadth-first, depth-first, and Discovery (which supports robots.txt and sitemaps).
• Scrapy Integration: Offers optional use of Scrapy for fetching and parsing tasks.

Once the frontier is running, the crawler interacts with it by requesting pages.

• Retrieving pages: Use the get_next_requests() method to ask the frontier for the next batch of pages.
• Iterations: A single call to get_next_requests() that returns a non-empty list of pages is considered one frontier iteration.
• Tracking iterations: You can access the current iteration count via the iteration attribute.

Frontera settings provide a global namespace of key-value mappings used to customize the behavior of all core components, including the FrontierManager, Middleware, and Backend.

Settings are accessed as attributes of the frontera.settings.Settings object. When implementing custom components like Middleware or Backend, you can access the settings through their from_manager class methods using the manager.settings attribute.