courlan

courlan is a Python library designed to act as a 'brain' for web crawling, scraping, and document management. It provides tools to enhance the quality of document collections by filtering out low-value pages, identifying content by language, and deduplicating URL collections at scale.

Key capabilities include:

• URL Validation & Filtering: Using check_url to validate, normalize, and filter URLs in a single step.
• Cleaning & Normalization: Using clean_url and normalize_url to fix messy URLs.
• Link Extraction: Using extract_links and filter_links for crawl-aware extraction from HTML.
• Sampling: Using sample_urls to select a specific number of URLs per domain.
• URL Management: Using UrlStore for domain-classified storage with visit tracking.
• Crawling Orchestration: Managing crawl delays, robots.txt compliance, and download scheduling.
• CLI Tools: A command-line interface for filtering and sampling URL files.

Setting strict=True in check_url enables a more rigorous filtering pipeline. This is useful for cleaning datasets for high-quality web crawling.

Key differences in Strict Mode:

• Query parameters: All parameters are removed except for a small allowlist (e.g., page, id, post).
• URL fragments: Stripped entirely.
• File extensions: Rejects non-web extensions (like .pdf, .zip) and uses pattern-based detection (e.g., /img/).
• Adult/Video content: Rejects URLs matching adult or video path patterns.
• Domain blacklist: Rejects URLs from major platforms/CDNs/social media listed in the internal BLACKLIST.
• Path filtering: Rejects landing pages (e.g., index, home) and utility paths (e.g., login, contact).
• Language detection: Uses both path segments and subdomains.

from courlan import check_url

# Blocked: major platform in the blacklist
check_url('https://www.twitter.com/', strict=True)
# None

# Aggressive query parameter trimming
check_url('https://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.org', strict=True)
# ('https://httpbin.org/redirect-to', 'httpbin.org')

# Rejected: adult content pattern
check_url('https://example.com/porn/page', strict=True)
# None

from courlan import check_url

# blocked in strict mode: major platform in the blacklist
check_url('https://www.twitter.com/', strict=True)
# None

# query parameters trimmed more aggressively
check_url('https://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.org', strict=True)
# ('https://httpbin.org/redirect-to', 'httpbin.org')

# adult content pattern rejected in strict mode
check_url('https://example.com/porn/page', strict=True)
# None

You can persist the UrlStore state to disk to resume crawls across sessions.

Warning: write() and load_store() use Python's pickle format. Only load files you have written yourself or that you trust, as loading malicious pickle files can execute arbitrary code.

Persistence Workflow:

1. Use store.write(filename) to save the current state.
2. Use load_store(filename) to restore a previously saved store.

Tips:

• Use compressed=True during initialization to reduce both memory usage and the on-disk footprint.
• For very large crawls, perform periodic saves and use incremental files (e.g., state-0001.db, state-0002.db) to avoid massive single writes and improve robustness.
• To export data for human inspection, use store.dump_urls() and write the resulting list to a file.

from courlan import UrlStore, load_store

store = UrlStore()
store.add_urls(['https://example.com/1', 'https://example.com/2'])
store.get_url('https://example.com')

# Save state to disk
store.write('crawler_state.db')

# Later: resume from disk
store = load_store('crawler_state.db')
print(store.get_unvisited_domains())

To build a crawler, use UrlStore to manage the queue of URLs. You can iterate through unvisited domains and retrieve a URL for each domain using get_url(domain), which marks that URL as visited. Use unvisited_websites_number() to control the loop termination.

Note: get_url(domain) returns a single URL for that domain. If you need to extract new links from a page, use store.add_from_html(html_content, base_url) to add them to the store.

from courlan import UrlStore

store = UrlStore(language='en', strict=True)
store.add_urls([
    'https://example.com/page1',
    'https://example.com/page2',
    'https://other.org/article',
])

while store.unvisited_websites_number() > 0:
    for domain in store.get_unvisited_domains():
        url = store.get_url(domain)  # marks as visited
        if not url:
            continue
        print(f"Visiting: {url}")
        # Example integration with requests:
        # response = requests.get(url, timeout=10)
        # store.add_from_html(response.text, url)

A typical crawl loop involves initializing the store with desired filters, seeding it with URLs, and iterating through unvisited domains while periodically checkpointing the state.

Best Practices:

• Keep frequently-accessed state in memory.
• Persist state periodically using write().
• Set language at construction time to filter URLs during the add phase rather than filtering them later.
• For long-running processes, periodically clear internal caches using courlan.meta.clear_caches() to manage memory.

from courlan import UrlStore

store = UrlStore(language='en', strict=True, compressed=True)
seed_urls = ['https://example.com']
store.add_urls(seed_urls)

while store.unvisited_websites_number() > 0:
    domains = store.get_unvisited_domains()
    if not domains:
        break
    
    url = store.get_url(domains[0])
    html = fetch(url)  # your downloader implementation
    
    # Extract and add new links found on the page
    store.add_from_html(html, url, external=False)
    
    # Periodically checkpoint state
    if should_checkpoint():
        store.write('crawler_state_checkpoint.db')