repository·main·Indexed 42 days ago
https://github.com/elastic/crawlerA lightweight, open-source, CLI-driven tool for discovering, extracting, and indexing web content directly into Elasticsearch via Docker. It supports on-demand or scheduled crawls, primary and purge crawl lifecycles for index freshness, and provides a CLI for tasks such as `crawler crawl`, `crawler schedule`, and `crawler urltest`. Compatible with Elasticsearch 8.x (v0.2.0+) and 9.x (v0.2.1+) on Linux and OSX.
Use this comparison to decide between the self-hosted Open Crawler and the cloud-based Elastic Crawler.
Key distinctions for Open Crawler users:
_bulk API for indexing.Both versions support advanced features like Ingest Pipelines, Binary Content Extraction, Crawl Rules (allow/disallow), Extraction Rules (CSS, XPath, URL selectors), Crawler Directives (robots.txt, sitemaps, etc.), Scheduling, and Event Logging in Elasticsearch.
The Open Crawler uses YAML configuration files to define crawl behavior and connection settings. There are two distinct types of configuration files:
crawler.yml (Required): Defines what to crawl, where to send results (sinks), and general crawl behavior. This is mandatory for all crawl jobs.elasticsearch.yml (Optional): Contains Elasticsearch connection settings. This is only required if your output sink is set to elasticsearch. It is useful for sharing connection settings across multiple crawlers.Precedence Rule: You can include Elasticsearch settings directly inside your crawler.yml. If you do, settings in the crawler.yml will override any settings found in a separate elasticsearch.yml file.
The crawler operates through a coordinated pipeline of components:
seed_urls from the config, creates crawl_task objects, and adds them to a queue.crawl_result objects containing discovered links.console or file: Outputs the result immediately after crawling.elasticsearch: Adds results to a bulk queue. When a threshold (queue count or byte size) is met, it flushes the queue using a _bulk API request to the configured Elasticsearch instance.Key Abstractions:
coordinator: Manages the crawl lifecycle and task queue.http_executor: Handles the actual network requests.output_sink: Manages where the data goes (Console, File, or Elasticsearch).document_mapper: Handles data transformation/formatting.Extraction rules allow you to customize how the crawler extracts specific content from webpages. They are configured within the domains[].extraction_rulesets field in your crawler configuration file.
domains[].url.url_filters array in the extraction_rulesets.url_filters array is empty, rules are applied to every result for that domain.url_filters array, the associated rules are executed.rules defined in that ruleset.Open Crawler utilizes Elasticsearch ingest pipelines to power content extraction features. These pipelines perform pre-processing on documents before they are ingested into the system.
By default, Open Crawler uses a pipeline named ent-search-generic-ingestion. This pipeline is automatically created when Elasticsearch starts for the first time. If you need to customize how documents are processed during ingestion, you can modify or create custom Elasticsearch ingest pipelines.
Crawl rules are instructions used to allow or deny specific paths within a domain. They are configured in the crawler's YAML configuration file under the crawl_rules key for a specific domain.
Key behaviors:
deny rule.Event logs are verbose, non-system-level logs that track specific crawler actions. Unlike system logs (which use DEBUG, INFO, etc.), event logs are tied to the lifecycle of a crawl.
Key events tracked include:
Format: Event logs are written in Elastic Common Schema (ECS) format, making them highly compatible with Elasticsearch filtering. They are useful for debugging why specific URLs were denied or how long specific tasks took.
{
"service.ephemeral_id":"6807937b2fe88a837f68cb36",
"service.type":"crawler",
"service.version":"0.3.0",
"process.pid":1283,
"host.name":"85811fb66e17",
"@timestamp":"2025-04-22T13:43:33Z",
"event.id":"68079d052fe88ab2087a5f17",
"process.thread.id":32728,
"crawler.crawl.id":"680793792fe88a837f68cb35",
"crawler.crawl.stage":"primary",
"url.full":"https://sfbay.craigslist.org/eby/apa/d/berkeley-charming-1br-in-excellent/7831506997.html",
"url.scheme":"https",
"url.domain":"sfbay.craigslist.org",
"url.path":"/eby/apa/d/berkeley-charming-1br-in-excellent/7831506997.html",
"crawler.url.hash":"823d1c5611fd8be2fd3faa1268d9712dd0174da3",
"event.type":"denied",
"event.action":"url-extracted",
"event.module":"html",
"event.outcome":"success",
"event.start":"2025-04-22T13:43:33Z",
"event.end":"2025-04-22T13:43:33Z",
"event.duration":3417,
"crawler.url.deny_reason":"content_type_denied",
"message":"Unexpected content type for a crawl task with type=content",
"event.kind":"event"
}The Elastic Open Web Crawler operates in two distinct stages to ensure index freshness and accuracy: a Primary crawl and a Purge crawl.
seed_urls and discovers new links. Each discovered URL is processed in a separate thread. Results are collected in a pool and indexed into Elasticsearch using the _bulk API once a configured threshold is met.last_crawled_at timestamp against the primary crawl's start time. Any document with a last_crawled_at value earlier than the start time is re-crawled.Cleanup: At the end of the purge crawl, any documents in the index that were neither updated during the primary crawl nor the purge crawl are deleted from Elasticsearch. This ensures the index only contains content currently reachable and valid according to your crawl rules and crawler directives.
Rules define the specific data to be pulled from a page. Rules are an array within an extraction_ruleset.
action)extract: Finds an element using a selector and adds its content to the document under field_name. If multiple elements are found, they are combined using join_as.set: Checks if an element matching the selector exists. If it does, the crawler adds the configured value to the document under field_name. If the element is missing, nothing is added.field_name: The name of the field in the resulting document where the data is stored. (Must not be a predefined field name from the document schema).selector: The method used to find content. The behavior depends on the source:html source: Supports CSS selectors or XPath selectors.url source: Must be a regular expression (regex). It is recommended to use capturing groups to specify the exact content to store.source: Specifies where to look. Supported values are html or url.join_as: (Only for extract action) Defines how to concatenate multiple found values. Values: string or array.value: (Only for set action) The value to insert if the selector finds a match. Cannot be null.URL filters determine which specific pages within a domain trigger an extraction ruleset. You define them using a type and a pattern.
Filter Types (type):
begins: Matches the start of the URL endpoint.ends: Matches the end of the URL endpoint.contains: Matches any value within the endpoint.regex: Matches using a regular expression.Example Patterns for http://example.com/blog/help/contact:
| Type | Pattern Example | Resulting Match |
|---|---|---|
begins | /blog | Matches |
ends | contact or /contact | Matches |
contains | help or /help/ | Matches |
regex | ^blog/help/support$ | No match (unless exact) |
Configuration files are processed using Embedded Ruby (ERB) template syntax before being parsed as YAML. This allows you to inject environment variables or define default values directly within your .yml files.
Use the <%= ... %> syntax to embed Ruby code.
Common Patterns:
Referencing an environment variable:
elasticsearch:
username: <%= ENV['ES_USER'] %>
password: <%= ENV['ES_PASS'] %>Providing a default value if an environment variable is missing:
output_path: <%= ENV['OUTPUT_PATH'] || '/tmp/crawl-output' %>elasticsearch:
username: <%= ENV['ES_USER'] %>
password: <%= ENV['ES_PASS'] %>
output_path: <%= ENV['OUTPUT_PATH'] || '/tmp/crawl-output' %>