DOCSEARCHbyAlgolia
Submit Docs

Apache StormCrawler Documentation

repository·main·Indexed 21 days ago

https://github.com/apache/stormcrawler

An open-source collection of Java resources for building low-latency, scalable web crawlers on top of Apache Storm. It includes modules for LLM-based content extraction via stormcrawler-ai, AWS integration (S3 and deprecated CloudSearch) via stormcrawler-aws, and OpenSearch indexing via stormcrawler-opensearch-java. The system requires an Apache Storm cluster and URLFrontier for seed URL management.

Tokens
33.5K
Snippets
97
Records
150
Agent score
76%

What's inside Apache StormCrawler

  1. Overview of Apache StormCrawler capabilities

    main

    Apache StormCrawler is a library and collection of reusable components built on Apache Storm for creating low-latency, scalable web crawlers. It is designed for scenarios where URLs arrive as continuous streams or for large-scale recursive crawls.

    Key architectural features include:

    • Pluggable Components: Uses Apache Storm Spouts and Bolts for modularity.
    • Distributed URL Management: Integrates with URLFrontier.
    • Document Parsing: Uses ParserBolt with Apache Tika support.
    • Indexing & Storage: Supports OpenSearch, Apache Solr, and WARC (Web ARChive) file formats.
    • Headless Crawling: Supports Playwright for rendering.
    • Filtering: Provides URL Filters (pre-fetch) and Parse Filters (post-fetch).
    • Fetchers: Robust HTTP fetching via Apache HttpComponents or OkHttp.

  2. Overview of stormcrawler-opensearch-java components

    main

    The stormcrawler-opensearch-java module provides several Apache Storm components for interacting with OpenSearch:

    • IndexerBolt: Used for indexing documents crawled by StormCrawler.
    • Spouts and StatusUpdaterBolt: Used for persisting URL information during recursive crawls.
    • MetricsConsumer and StatusMetricsBolt: Used for sending URL status breakdowns as metrics to visualize evolution over time.

    Note on Index Schemas: Schemas are automatically created by the bolts upon their first use. If you require custom index definitions, you should provide them manually before starting the Storm topology.

  3. Use StormCrawler Solr resources for indexing and persistence

    main

    The stormcrawler-solr module provides several specialized components for building Storm topologies that interact with Apache Solr collections:

    • IndexerBolt: An implementation of AbstractIndexerBolt used to index parsed data and metadata into a specific Solr collection.
    • MetricsConsumer: A class used to store Storm metrics within Solr.
    • SolrSpout: A Spout that retrieves URLs from a specified Solr collection to drive the crawling process.
    • StatusUpdaterBolt: An implementation of AbstractStatusUpdaterBolt used to store the status of each URL and its serialized metadata in Solr.

  4. Use metadata to control HTTP request behavior

    main

    StormCrawler's HTTPProtocol implementations can change their request behavior based on keys present in the metadata object. This allows for fine-grained, per-link control over headers, methods, and proxies.

    Common Metadata Keys for HTTP Control

    KeyBehavior
    last-modifiedUses the value for the If-Modified-Since header.
    protocol.etagUses the value for the If-None-Match header.
    http.acceptOverrides the Accept header (v1.11+).
    http.accept.languageOverrides the Accept-Language header (v1.11+).
    protocol.set-cookieIf http.use.cookies is true, sends cookies from the previous response if they match the domain.
    http.method.headSends a HEAD request (v1.12+ for httpclient).
    http.post.jsonSends a POST request (v1.12+ for okhttp).
    protocol.set-headersAdds custom headers to the request.
    http.proxy.skipIf true, bypasses all proxy managers for this request.
    http.proxyUses a full connection string (e.g., http://user:pass@proxy.example.com:8080) as the proxy.
    http.proxy.host, http.proxy.port, etc.Builds a per-request proxy using individual components.

    Custom Headers

    To add custom headers, use the protocol.set-headers key with a list of header=value strings.

    "protocol%2Eset-header": [
  "header1=value1",
  "header2=value2"
]

  5. Compare FetcherBolt and SimpleFetcherBolt

    main

    StormCrawler provides two types of fetcher bolts:

    • FetcherBolt: Uses internal queues organized by hostname/domain/IP (fetcher.queue.mode) and multiple FetchingThreads (fetcher.threads.number). It enforces politeness by delaying requests to the same server (fetcher.server.delay). It is designed for high-throughput, multi-threaded fetching.
    • SimpleFetcherBolt: Does not use internal queues or multi-threading. It fetches tuples directly in its execute method. To achieve scale, you must declare multiple instances of this bolt in your topology and use a URLPartitioner to distribute URLs.

  6. How the Status Stream works

    main

    StormCrawler uses two Apache Storm streams: the _default_ stream and the _status_ stream.

    • _default_ stream: Carries the URL being processed and its content/metadata. It is typically used at the end of the pipeline by an indexing bolt (e.g., OpenSearch, HBase).
    • _status_ stream: Passes information about URLs to a persistence layer. This is critical for recursive crawls. A bespoke bolt (extending AbstractStatusUpdaterBolt) consumes this stream to update storage, which a Spout then uses to feed new URLs back into the topology.

    Parsing bolts emit to the status stream to handle outlinks or notify of problems (e.g., unparsable content). Fetching bolts use it to handle redirections, exceptions, or unsuccessful HTTP statuses.

  8. StormCrawler Core Terminology

    main

    Understanding these concepts is essential for configuring and managing StormCrawler:

    • Topology: The overall data processing graph in Storm, consisting of spouts and bolts.
    • Spout: A source component in a topology that emits streams of data.
    • Bolt: A component that processes, transforms, or routes data streams.
    • Flux: A declarative configuration framework that allows defining Storm topologies using YAML files instead of Java code.
    • Frontier: The component responsible for managing and prioritizing the list of URLs to be fetched.
    • Seed: The initial URL(s) from which the crawler starts discovery.

  9. Use the Selenium protocol for dynamic web crawling

    main

    The Selenium protocol allows StormCrawler to interact with dynamic, JavaScript-heavy web pages by using a Selenium WebDriver. This is essential when standard crawling methods cannot execute the JavaScript required to render page content.

    ⚠️ Deprecated: The Selenium module is deprecated and will be removed in the next major release of StormCrawler.

  10. Use PageActions to customize post-navigation behavior

    main

    You can inject custom logic (like clicking tabs, scrolling, or dismissing banners) after a page loads but before content is captured by using a PageAction chain.

    1. Create a JSON configuration file defining the chain of actions.
    2. Reference this file in your StormCrawler configuration using playwright.page.actions.config.file.

    Actions are loaded in order. If one action fails, it is logged and swallowed so the rest of the chain can continue. Failures in configure() (validation) will stop the topology from starting.

    {
  "org.apache.stormcrawler.protocol.playwright.PageActions": [
    {
      "class": "org.apache.stormcrawler.protocol.playwright.actions.ExpandClickablesAction",
      "name": "tabs",
      "params": {
        "selectors": [".tab-widget .tab-header"],
        "root": ".tab-widget",
        "body": ".tab-widget-body",
        "waitMs": 300
      }
    }
  ]
}

  11. Monitor crawl progress with OpenSearch Dashboards

    main

    After importing the dashboards, you can monitor your crawl via the OpenSearch Dashboards UI:

    • Crawl Status Dashboard: Displays tables containing the count of URLs per status and the top hostnames per URL count.
    • Crawl Metrics Dashboard: Used to monitor real-time progress (e.g., active threads, pages fetched, bytes per second).

    Note: The storm.ndjson file can be used to display Storm's internal metrics, but it is not added by default.

  12. Understand the URL Status Lifecycle and Retries

    main

    StormCrawler uses a status-based system to drive scheduling and retries:

    • DISCOVERED: New URL, not yet fetched.
    • FETCHED: Successfully retrieved and processed.
    • FETCH_ERROR: Transient error (e.g., network timeout, HTTP 5xx). This status triggers a retry.
    • ERROR: Terminal error (e.g., too many retries, HTTP 4xx). Not retried by default.
    • REDIRECTION: HTTP 3xx redirect encountered.

    Retry Mechanism: When a URL hits FETCH_ERROR, the AbstractStatusUpdaterBolt increments a fetch.error.count in the metadata. Once this reaches max.fetch.errors (default: 3), the status is escalated to ERROR.

    Default Scheduling Intervals:

    • FETCHED: fetchInterval.default (default: 1440 minutes).
    • FETCH_ERROR: fetchInterval.fetch.error (default: 120 minutes).
    • ERROR: fetchInterval.error (default: -1, never refetch).