DOCSEARCHbyAlgolia
Submit Docs

Wayback Machine Save Page Now (SPN) Scripts

repository·main·Indexed 41 days ago

https://github.com/overcast07/wayback-machine-spn-scripts

A collection of Bash scripts, featuring spn.sh, designed to automate sending web page capture requests to the Internet Archive's Save Page Now (SPN) service. It supports parallel capture jobs, recursive outlink management via regex patterns, rate limiting, and session resumption. Compatible with Windows 10 (via WSL), macOS, Linux, and BSD.

Tokens
1.7K
Snippets
5
Records
8
Agent score
50%

What's inside wayback-machine-spn-scripts

  1. Overview of Wayback Machine Save Page Now Bash scripts

    main

    The wayback-machine-spn-scripts repository provides spn.sh, a Bash script designed to interact with the Internet Archive Wayback Machine's Save Page Now (SPN) service. It allows users to request that live web pages be saved to the archive.

    Key Features

    • Cross-platform compatibility: Works on Windows 10 (via WSL), macOS, Linux, and BSD.
    • Parallelism: Ability to run multiple capture jobs in parallel.
    • Resilience: Includes automatic retrying, error handling, and the ability to resume aborted sessions.
    • Advanced Outlink Management: Supports recursive and selective saving of outlinks, including the ability to include or exclude specific URL groups (e.g., avoiding login forms).
    • Rate Limiting: Unlike the native SPN outlinks function, this script allows limiting parallel jobs to avoid overwhelming target websites.
    • Authentication: Supports optional API authentication.
    • Extensibility: The script uses POST requests to retrieve rich JSON response data, making it easy to extend for custom data processing.

  2. Understand the data folder and files

    main

    The script stores logs and state in a sub-folder of ~/spn-data. You can customize this with -f (folder) and -i (suffix).

    Control Files (Can be modified manually to affect operation)

    • failed.txt: URLs that failed and will be retried. Add/remove URLs here.
    • outlinks.txt: Compiled matching outlinks. Add/remove URLs here.
    • index.txt: Records which links are part of the current list. Used for resuming sessions.
    • include_pattern.txt / exclude_pattern.txt: Contain the regex patterns used for -o and -x.
    • max_parallel_jobs.txt: Sets the limit for -p.
    • status_rate.txt: Sleep time between status checks.
    • list_update_rate.txt: Sleep time between list updates (from -t).
    • capture_job_rate.txt: Sleep time between jobs (from -w).
    • quit.txt: If this file exists, the script will exit after finishing the current list.

    Status and Lock Files

    • lock.txt: Created when a rate limit is hit; prevents new jobs from starting until resolved.
    • daily_limit.txt: Created when the archive.org daily limit is reached. The script pauses until the next hour.

    Log Files (Read-only/Informational)

    • success.log: Submitted URLs for successful jobs.
    • captures.log: The resulting Wayback Machine URLs.
    • error-json.log: JSON responses for unsuccessful jobs.
    • failed.log: URLs that failed and won't be retried, with error codes.
    • invalid.log: Jobs that failed to start.

    Note on JSON logs: Log files containing JSON are not valid JSON files themselves. To convert a .log file to a valid .json file, use:

    sed -e 's/}$/},/g' <<< "[$(<filename.log)]" > filename.json

  4. Basic usage of spn.sh

    main

    The script accepts either a single URL, multiple URLs, or a text file containing one URL per line as its first argument. If a file is provided, the script processes the URLs within it.

    Submit multiple URLs:

    ./spn.sh https://example.com/page1/ https://example.com/page2/

    Submit URLs from a text file:

    ./spn.sh urls.txt
    spn.sh urls.txt

  5. Resume an aborted session

    main

    If a session is interrupted, you can resume it using the -r flag followed by the path to the data folder used in the previous session.

    Requirements for resuming:

    • index.txt and success.log must exist in the folder.
    • If outlinks.txt exists, the script will also account for those links and the contents of captures.log.

    Note: Settings (like -o or -x) are not carried over from the previous command, except for outlinks options which are read from the existing files. To avoid collecting new outlinks during a resume, use -x '.*'.

    Example:

    ./spn.sh -r ~/spn-data/20231027120000_mysite
    spn.sh -r /path/to/data_folder

  6. Save recursive outlinks with regex patterns

    main

    The script can discover and archive outlinks found on captured pages using POSIX ERE regular expressions.

    • -o pattern: Save outlinks matching the regex pattern.
    • -x pattern: Save outlinks that do not match the regex pattern.
    • -o '': A shortcut to save all detected outlinks.

    Example: Save all outlinks:

    spn.sh -o '' https://example.com/

    Example: Save YouTube or Reddit, but exclude Facebook:

    spn.sh -o 'youtube|reddit' -x 'facebook' https://example.com/

    Example: Save specific subdomain:

    spn.sh -o 'https?://fr\.wikipedia\.org(/|$)' https://example.com/
    spn.sh -o 'youtube|reddit' -x 'facebook' https://example.com/

  7. Run capture jobs in parallel or with delays

    main

    You can control the concurrency and timing of capture jobs using the -p and -w flags.

    • -p N: Run at most N capture jobs in parallel (default: 20). Range: 1-60.
    • -w N: Wait at least N seconds after starting a capture job before starting another (default: 2.5s).

    Example: High concurrency (15 parallel jobs):

    spn.sh -p 15 urls.txt

    Example: Sequential jobs with 60s delay:

    spn.sh -p 1 -w 60 urls.txt
    spn.sh -p 15 urls.txt

  8. Reference: spn.sh command line flags

    main

    All flags should be placed before arguments. If a string contains characters that need to be escaped in Bash, wrap it in quotes.

    FlagArgumentDescription
    -aauthS3 API keys in accesskey:secret format. Enables capture_outlinks=1 and capture_screenshot=1 via -d.
    -cargsPass additional arguments to curl (e.g., -c '-x socks5h://127.0.0.1:9150/' for Tor).
    -ddataCapture request options/POST data (e.g., -d 'force_get=1&if_not_archived_within=86400').
    -ffolderUse a custom data folder location.
    -isuffixAdd a suffix to the data folder name (ignored if -f is used).
    -n(none)Tell Save Page Now not to save error pages.
    -opatternSave detected capture outlinks matching regex (ERE) pattern.
    -pNRun at most N capture jobs in parallel (default: 20).
    -q(none)Discard JSON for completed jobs to save disk space.
    -rfolderResume with the remaining URLs of an aborted session.
    -s(none)Use HTTPS for all captures and convert HTTP input URLs to HTTPS.
    -tNWait at least N seconds before updating the main URL list (default: 3600).
    -wNWait at least N seconds after starting a job before starting another (default: 2.5).
    -xpatternSave detected capture outlinks NOT matching regex (ERE) pattern.