DOCSEARCHbyAlgolia
Submit Docs

waybackurls Documentation

repository·master·Indexed 26 days ago

https://github.com/tomnomnom/waybackurls

A CLI tool that retrieves all known URLs associated with a list of domains from the Wayback Machine, Common Crawl, and VirusTotal. It supports input via stdin or command-line arguments and provides flags for filtering subdomains, including fetch dates, and retrieving specific crawled versions via the Wayback Machine CDX API.

Tokens
418
Snippets
2
Records
6
Agent score
39%

What's inside waybackurls

  2. Use waybackurls to fetch URLs from the Wayback Machine

    master

    The waybackurls tool accepts line-delimited domains via stdin. It fetches all known URLs from the Wayback Machine for *.domain and outputs the results to stdout.

    To use it, pipe a list of domains into the command and redirect the output to a file.

    cat domains.txt | waybackurls > urls

  4. Use the --get-versions flag

    master
    When using the --get-versions flag, waybackurls queries the Wayback Machine CDX API to find specific crawled versions of the provided URL. The output format changes to a list of Wayback Machine archive links (e.g., https://web.archive.org/web/[timestamp]if_/[url]).

  5. Configure waybackurls CLI flags

    master

    The following flags are available to modify the output of waybackurls:

    • --dates: When set, the output includes the fetch date in the first column (formatted in RFC3339).
    • --no-subs: Prevents the inclusion of subdomains of the target domain.
    • --get-versions: Instead of listing all URLs, this mode lists URLs for crawled versions of the input URL(s).

  6. Use waybackurls CLI to fetch URLs

    master

    The waybackurls CLI tool retrieves historical URLs for a given domain from various sources including the Wayback Machine, Common Crawl, and VirusTotal.

    It accepts domains via command-line arguments or via stdin. If no arguments are provided, it reads domains from standard input.