cdx_toolkit

You can install cdx_toolkit via pip. For remote filesystem support (like S3), install the [s3] extra. To install all optional dependencies, use the [all] extra.

$ pip install cdx_toolkit
$ pip install cdx_toolkit[s3]   # enable S3 and other remote filesystem support
$ pip install cdx_toolkit[all]  # install all optional dependencies

The cdxt command-line tool allows you to iterate over CDX captures, create WARC files from captures, and estimate the number of available results for a given URL. It supports querying both Common Crawl and Internet Archive CDX indices.

# Example: Iterate over captures for a URL and output as JSONL
cdxt iter https://example.com --jsonl

# Example: Create a WARC file for a URL
cdxt warc https://example.com --prefix MY_COLLECTION

# Example: Get an estimate of available results
cdxt size https://example.com

You can use cdx_toolkit programmatically by instantiating CDXFetcher.

• source='cc' targets Common Crawl.
• source='ia' targets the Internet Archive.
• get_size_estimate(url) returns an estimate of the number of captures.
• iter(url, limit=N) returns an iterator of capture objects.
• Capture objects contain metadata and a .content attribute for the raw bytes of the web capture.

import cdx_toolkit

cdx = cdx_toolkit.CDXFetcher(source='cc')
url = 'commoncrawl.org/*'

# Get size estimate
print(url, 'size estimate', cdx.get_size_estimate(url))

# Iterate through captures
for obj in cdx.iter(url, limit=1):
    print(obj)
    print(obj.content)  # Access raw bytes

When working with cdx_toolkit, understand these core concepts:

• Capture: A single crawled URL (webpage, redirect, error, or revisit record).
• Timestamp: A string of digits representing YYYYMMDDHHMMSS (UTC). Used with --from, --to, or from_ts=, to=, and closest=.
• URLKey: A SURT (a munged URL for deduplication/sorting).
• Digest: A SHA1 checksum of the capture content used to identify identical content.
• Revisit Record: A record indicating the page is identical to a previous capture (status is often - rather than 200).
• Wildcards:
  • *.example.com (matchType='domain'): Matches example.com and all subdomains.
  • example.com/*: Matches any path on example.com.

Unlike Common Crawl, the Internet Archive (IA) uses a single continuous index. To select a specific period for IA, you must use the --from and --to flags with timestamps in YYYYMMDDHHMMSS format (UTC). If no range is specified, it defaults to the most recent year.

$ cdxt --ia --limit 1 --from 2008 --to 200906302359 iter 'commoncrawl.org/*'

Common Crawl data is organized into individual monthly or yearly crawls. Use the --crawl flag to target specific timeframes or specific crawl identifiers.

• --crawl CC-MAIN-2024-33: A single specific crawl.
• --crawl 3: The latest 3 crawls.
• --crawl CC-MAIN-2018: All crawls from the year 2018.
• --crawl CC-MAIN-2018,CC-MAIN-2019: A specific list of crawls.

$ cdxt --crawl CC-MAIN-2024-33 iter 'commoncrawl.org/*'
$ cdxt --crawl 3 --limit 10 iter 'commoncrawl.org/*'
$ cdxt --crawl CC-MAIN-2018,CC-MAIN-2019 iter 'commoncrawl.org/*'

The cdxt CLI tool allows you to interact with Common Crawl (CC) and Internet Archive (IA) CDX indices. It supports commands like iter (iterate over records), warc (extract to WARC files), and size (get size estimates).

Important Notes:

• Argument Order: Switches must be placed either before or after the command (iter, warc, or size).
• Default Limits: To prevent massive queries, defaults include a maximum of 1000 records (--limit 1000) and a 1-year lookback for CC. Use --limit, --crawl, --from, or --to to override these.
• Verbosity: Use -v or -vv to see debug information.

$ cdxt --cc size 'commoncrawl.org/*'
$ cdxt --cc --limit 10 iter 'commoncrawl.org/*'
$ cdxt --cc --limit 10 --filter '=status:200' iter 'commoncrawl.org/*'
$ cdxt --ia --limit 10 warc 'commoncrawl.org/*'
$ cdxt -vv --cc size 'commoncrawl.org/*'

The warc subcommand iterates over capture content and bundles them into a WARC file. It includes filtering capabilities to include or exclude specific URLs based on pattern matching.

cdxt warc <url> [options]

Options:
  --prefix <prefix>      Prefix for the WARC filename (default: TEST)
  --subprefix <subprefix> Subprefix for the WARC filename
  --size <int>            Target for the WARC filesize in bytes
  --creator <creator>     Creator of the WARC (person, organization, service)
  --operator <operator>   A person, if the creator is an organization
  --url-fgrep <pattern>  Pattern must be present in the URL to include it
  --url-fgrepv <pattern>  Pattern must NOT be present in the URL to include it
  --warc-download-prefix <prefix> Prefix for downloading content

The size subcommand provides an imprecise count of how many results are available for a given URL.

cdxt size <url> [options]

Options:
  --details              Show details of each subindex

The iter subcommand prints capture information for a specific URL. You can specify which fields to display and the output format.

cdxt iter <url> [options]

Options:
  --all-fields           Print all available fields
  --fields <fields>      Comma-separated list of fields (default: 'url,status,timestamp')
  --jsonl                Output in JSON Lines format
  --csv                  Output in CSV format