DOCSEARCHbyAlgolia
Submit Docs

waybackpack

repository·master·Indexed 25 days ago

https://github.com/jsvine/waybackpack

A Python-based command-line tool used to download entire archives of a specific URL from the Wayback Machine to a local directory. It supports date filtering via --from-date and --to-date, listing available snapshots with --list, and various configuration options for network throttling, error handling, and download behavior such as --raw and --uniques-only.

Tokens
1.5K
Snippets
4
Records
8
Agent score
35%

What's inside waybackpack

  2. Download Wayback Machine archives with waybackpack

    master

    Use waybackpack to download all archived snapshots of a URL within a specific timeframe to a local directory.

    To download snapshots up to a specific date, use the --to-date flag with a timestamp in YYYYMMDDhhss format (trailing digits are optional, e.g., 201604). Use the -d or --dir flag to specify the destination directory.

    waybackpack http://www.dol.gov/ -d ~/Downloads/dol-wayback --to-date 1996

  4. Reference: waybackpack CLI options

    master

    The waybackpack CLI accepts the following arguments and options:

    Positional Arguments

    • url: The URL of the resource you want to download.

    Required/Mutually Exclusive Options

    • -d DIR, --dir DIR: Directory to save the files. Creates the directory if it doesn't exist.
    • --list: Instead of downloading, only print the list of snapshots.

    Date Filtering

    • --from-date FROM_DATE: Earliest snapshot to download. Format: YYYYMMDDhhss (e.g., 201501).
    • --to-date TO_DATE: Latest snapshot to download. Format: YYYYMMDDhhss (e.g., 201604).

    Download Behavior

    • --raw: Fetch file in its original state, without processing by the Wayback Machine or waybackpack.
    • --root ROOT: The root URL from which to serve resources. Default: https://web.archive.org.
    • --uniques-only: Download only the first version of duplicate files.
    • --collapse COLLAPSE: An archive.org collapse parameter (e.g., --collapse timestamp:6).
    • --no-clobber: If a file is already present and >0 filesize, don't download it again.
    • --follow-redirects: Follow redirects.
    • --ignore-errors: Don't crash on non-HTTP errors; log and continue.
    • --max-retries MAX_RETRIES: Number of retries for 4XX or 5XX status codes.

    Network and Throttling

    • --user-agent USER_AGENT: User-Agent header. Default: waybackpack. It is recommended to include 'waybackpack' and an email address.
    • --delay DELAY: Sleep X seconds between each fetch.
    • --delay-retry DELAY_RETRY: Sleep X seconds between each post-error retry.

    Output and Logging

    • --quiet: Don't log progress to stderr.
    • --progress: Print a progress bar (requires tqdm). Mutes default logging.
    usage: waybackpack [-h] [--version] (-d DIR | --list) [--raw] [--root ROOT]
                   [--from-date FROM_DATE] [--to-date TO_DATE]
                   [--user-agent USER_AGENT] [--follow-redirects]
                   [--uniques-only] [--collapse COLLAPSE] [--ignore-errors]
                   [--max-retries MAX_RETRIES] [--no-clobber] [--quiet]
                   [--progress] [--delay DELAY] [--delay-retry DELAY_RETRY]
                   url

  5. Configure waybackpack download behavior

    master

    Use the following flags to control how waybackpack fetches and saves files:

    FlagDescription
    --rawFetch file in its original state, without any processing by the Wayback Machine or waybackpack.
    --root <url>The root URL from which to serve snapshotted resources. Default: {DEFAULT_ROOT}.
    --from-date <timestamp>Earliest snapshot to download (format YYYYMMDDhhss, e.g., 201501).
    --to-date <timestamp>Latest snapshot to download (format YYYYMMDDhhss, e.g., 201604).
    --uniques-onlyDownload only the first version of duplicate files.
    --no-clobberIf a file is already present (and >0 filesize), don't download it again.
    --follow-redirectsFollow redirects.
    --collapse <pattern>An archive.org collapse parameter (e.g., --collapse timestamp:6). Can be used multiple times.

  6. Configure waybackpack network and error handling

    master

    Use these flags to manage network requests, retries, and error tolerance:

    FlagDescription
    --user-agent <string>The User-Agent header. It is recommended to include 'waybackpack' and your email. Default: {DEFAULT_USER_AGENT}.
    --max-retries <int>Number of times to try accessing content with 4XX or 5XX status code before skipping. Default: 3.
    --delay <int>Sleep X seconds between each fetch. Default: 0.
    --delay-retry <int>Sleep X seconds between each post-error retry. Default: 5.
    --ignore-errorsDon't crash on non-HTTP errors (e.g., ChunkedEncodingError). Log error and continue.
    --quietDon't log progress to stderr.

  7. Configure waybackpack output and progress

    master

    Control how the CLI displays information during execution:

    FlagDescription
    --progressPrint a progress bar and mute default logging. Requires tqdm to be installed.
    --quietMutes progress logging.
    --versionShow the version number and exit.

  8. Use the waybackpack CLI to download or list snapshots

    master

    The waybackpack command allows you to download resources from the Wayback Machine to a local directory or simply list the available snapshot URLs.

    Basic Usage:

    To download files, you must provide a target url and a destination directory using --dir or --list to print snapshots instead of downloading.

    Required Arguments:

    • url: The URL of the resource you want to download.

    Mutually Exclusive Options (One required):

    • -d, --dir <directory>: The directory to save the files. This directory will be created if it doesn't exist.
    • --list: Instead of downloading, this prints the list of snapshot URLs to stdout.