tldextract Documentation

Public Suffix List

tldextract uses the Public Suffix List (PSL) to distinguish between public suffixes (e.g., .com, .co.uk) and private suffixes (e.g., blogspot.com, github.io).

Default Behavior

• Private Domains: By default, private suffixes are treated as regular domains. To treat them as suffixes, set include_psl_private_domains=True in TLDExtract.
• Caching: On first use, tldextract fetches the latest PSL and caches it indefinitely in $HOME/.cache/python-tldextract.

Install the tldextract package using pip to accurately separate a URL's subdomain, domain, and public suffix using the Public Suffix List (PSL).

pip install tldextract

You can set the cache directory for tldextract using the TLDEXTRACT_CACHE environment variable.

export TLDEXTRACT_CACHE="/path/to/cache"

The DiskCache implementation includes several safety features:

1. File Locking: It uses filelock to ensure that multiple processes do not corrupt the cache files during simultaneous read/write operations.
2. Unique File Extensions: Files are saved with the .tldextract.json extension. This prevents the clear() method from accidentally deleting files that do not belong to the tldextract cache if the cache_dir is incorrectly configured.
3. Namespace Isolation: Data is organized into subdirectories based on the provided namespace to prevent key collisions across different parts of an application.

Because tldextract is very lenient and will attempt to extract domains from almost any string, you may want to validate URLs using urllib.parse.urlsplit first, then use extract_urllib().

from urllib.parse import urlsplit
import tldextract

extract = tldextract.TLDExtract()
split_url = urlsplit("https://example.com/path")
result = extract.extract_urllib(split_url)

You can control where tldextract stores its cache using the TLDEXTRACT_CACHE environment variable.

If this variable is not set, the library follows these fallback rules:

1. It attempts to follow the XDG standard (using XDG_CACHE_HOME or ~/.cache/python-tldextract/<unique_identifier>).
2. If XDG is unavailable, it falls back to a .suffix_cache directory within the package directory itself.

You can instruct the CLI to use a specific TLD definition source using the --suffix_list_url flag. This flag accepts either a remote URL or a path to a local file. If a local file path is provided, the CLI will automatically convert it to a file URI.

You can provide multiple sources by repeating the flag.

Example: Using a local file

tldextract --suffix_list_url /path/to/custom_list.txt example.com

Example: Using a remote URL

tldextract --suffix_list_url https://example.com/list.txt example.com

The DiskCache class provides a disk-based cache mechanism specifically for values that can be serialized to JSON. It uses file locking to prevent race conditions during concurrent access.

Key Methods

• get(namespace: str, key: str | dict[str, Hashable]) -> object: Retrieves a value. Raises KeyError if the cache is disabled or the key is not found.
• set(namespace: str, key: str | dict[str, Hashable], value: object) -> None: Saves a value to the cache. If the cache is disabled, this is a no-op.
• clear() -> None: Deletes all files in the cache directory that end with the .tldextract.json extension or .tldextract.json.lock.