DOCSEARCHbyAlgolia
Submit Docs

Heritrix3 Documentation

repository·master·Indexed 25 days ago

https://github.com/internetarchive/heritrix3

An open-source, extensible, web-scale, archival-quality web crawler developed by the Internet Archive. Documentation covers installation via Docker Compose, crawl politeness guidelines, job configuration using CXML and profiles, and a comprehensive REST API for managing engine status, job lifecycles (create, build, launch, pause, terminate), and checkpointing.

Tokens
16.1K
Snippets
39
Records
85
Agent score
82%

What's inside Heritrix3

  4. Define Crawl Scope using DecideRules

    master

    The crawl scope determines which URIs are captured by a crawl. This is managed by a sequence of DecideRule objects. Each rule evaluates a URI and returns one of three decisions:

    • ACCEPT: The URI is ruled in.
    • REJECT: The URI is ruled out.
    • PASS: The rule has no opinion; the URI retains its previous status.

    A URI starts with no assumed status. Rules are applied in order. A URI is considered "in scope" only if its final status is ACCEPT. If its status is REJECT, it is discarded.

    Best Practices:

    • Start with recommended default configurations.
    • Perform small test crawls to understand how rules interact.
    • Make small, individual changes to the scope rather than creating a ruleset from scratch to avoid breaking the crawler's ability to make progress.

  5. Securely Access the Heritrix Web UI via SSH Tunneling

    master

    To maintain security, it is recommended to keep the Heritrix Web UI bound to localhost only. If Heritrix is running on a remote machine (e.g., crawler.example.com), you can access the Web UI locally using an SSH tunnel.

    Run the following command from your local machine to map local port 9999 to the remote port 8443:

    ssh -L localhost:9999:localhost:8443 crawloperator@crawler.example.com -N

    You can then access the interface at https://localhost:9999/.

  6. Trigger actions in a running crawl via the Action Directory

    master

    You can influence a running crawl by placing files into the job's action directory. Heritrix monitors this directory at regular intervals. Files are processed and then moved to a done directory.

    Supported File Suffixes:

    • .seeds: Adds new seeds to the running crawl.
    • .recover: Acts as a traditional recovery journal to reproduce queue state.
    • .include: Acts as a recovery journal that marks all listed URIs as 'already included' to suppress re-crawling.
    • .schedule: Inserts URIs into the crawling queues.
    • .force: Guarantees that listed URIs are re-enqueued and re-crawled, even if they were previously marked as included.

    Note: Files can be gzipped. You can insert .s. before the functional suffix (e.g., frontier.s.recover.gz) to force URIs to undergo scope-testing before insertion.

  7. Restart a crawl from a checkpoint via Web UI

    master

    If you want to resume a crawl from a specific saved state using the Web User Interface (WUI), follow these steps:

    1. Checkpoint: Click the "checkpoint" button on the running job page.
    2. Terminate: Once the checkpoint completes (wait for the notification), click "terminate".
    3. Teardown: Click the "teardown" button.
    4. Re-build: Click the "build" button. A dropdown menu containing previous checkpoint names will appear.
    5. Select: Choose the desired checkpoint from the dropdown.
    6. Launch: Click "launch".
    7. Unpause: Click "unpause" to begin crawling.

  8. Use Sheets for Site-specific Overrides

    master

    Sheets allow you to replace default settings on a per-domain or per-path basis using SURT Prefix syntax. This is useful for applying different crawling policies (e.g., a 'less polite' policy) to specific domains where you have explicit permission.

    To implement a Sheet, you must:

    1. Configure a SheetOverlaysManager.
    2. Define a SurtPrefixesSheetAssociation to map specific URL patterns (using SURT syntax) to target sheet names.
    3. Define the Sheet itself, containing a map of property-paths to new values.

    Warning: Use less-polite settings only if you have explicit permission from the target site. Always include contact information in your User-Agent string.

    sheetOverlaysManager(SheetOverlaysManager) {
    bean.autowire = 'byType'
}

lessPoliteAssociation(SurtPrefixesSheetAssociation) {
    surtPrefixes = [
        'http://(com,example,www,)/',
        'http://(com,example1,www,)/',
    ]
    targetSheetNames = ['lessPolite']
}

lessPolite(Sheet) {
    map = [
        'disposition.delayFactor': '0.0',
        'disposition.maxDelayMs': '0',
        'disposition.minDelayMs': '0',
        'queueAssignmentPolicy.parallelQueues': '5',
    ]
}

  11. Run Heritrix using Docker

    master

    Heritrix can be run as a Docker container. The Web UI is enabled by default on port 8443. When using Docker, command-line options are exposed via environment variables instead of direct flags.

    mkdir jobs
docker run --init --rm -d -p 8443:8443 -e "USERNAME=admin" -e "PASSWORD=admin" -v $(pwd)/jobs:/opt/heritrix/jobs iipc/heritrix

  12. Crawl Politeness and Identification Best Practices

    master

    When operating Heritrix, follow these politeness guidelines to minimize impact on seed sites:

    • Respect Directives: Adhere to robots.txt exclusion directives and META nofollow tags.
    • Set Politeness Policies: Configure your crawl to manage the load placed on target servers.
    • Identify Your Crawler: Always include contact information in your User-Agent string so site administrators can contact you or adapt their server behavior if your crawl causes issues.