DOCSEARCHbyAlgolia
Submit Docs

FastWARC

repository·develop·Indexed 40 days ago

https://github.com/chatnoir-eu/chatnoir-resiliparse

A high-performance WARC parsing library written in Rust with Python bindings. Part of the ChatNoir Resiliparse collection, it provides robust utilities for parsing, extraction, and resource-guarded batch processing of web archive data. Version 1.0.9 supports uncompressed, Gzip, LZ4, and Zstandard formats.

Tokens
58.4K
Snippets
201
Records
260
Agent score
100%

What's inside FastWARC

  1. Overview of Resiliparse Extraction Utilities

    develop

    The Extraction Utilities module is a performance-optimized library designed to extract structural or semantic information from noisy, raw web data. It is intended for downstream processing tasks such as:

    • Content Extraction: Extracting main content or removing boilerplate.
    • Schema Extraction: Identifying and pulling structured data from web pages.
    • Data Cleansing: General cleaning and normalization of web data.

    The module is optimized for high-throughput processing of raw web data.

  2. Overview of FastWARC (Python)

    develop

    FastWARC is a high-performance WARC parsing library written in Rust with Python bindings. It is part of the ChatNoir Resiliparse toolkit designed for fast and robust web data processing.

    Key Features:

    • Supports compressed and uncompressed WARC/1.0 and WARC/1.1 streams.
    • Supported compression algorithms: Gzip, Zstd, and LZ4.

  4. Overview of ChatNoir Resiliparse modules

    develop

    ChatNoir Resiliparse is a collection of tools for parsing and analyzing web archive data. The toolkit is divided into two primary modules:

    1. Resiliparse: A core module containing:
      • Parsing Utilities: Tools for handling encodings, raw protocol payloads, and HTML parsing/extraction.
      • Process Guard: Decorators and context managers used to limit execution time and memory usage for individual tasks in batch processing jobs.
      • Itertools: Helper functions for robustly iterating over data from unreliable sources.
    2. FastWARC: A high-performance WARC parsing library for Python (implemented in C++/Cython). It supports compressed (Gzip, LZ4) and uncompressed WARC/1.0 and WARC/1.1 streams. It provides both a Python API and a CLI.

  5. Overview of Resiliparse modules

    develop

    The Resiliparse collection consists of two main modules:

    1. Resiliparse

    A core module containing several subcomponents:

    • Parsing Utilities: Optimized tools for handling encodings, detecting content types, parsing HTML, and language detection.
    • Extraction Utilities: Tools for extracting structural or semantic information (e.g., content extraction, boilerplate removal, schema extraction, and data cleansing).
    • Process Guards: Decorators and context managers used to limit execution time and memory usage for processing tasks, preventing individual failures from crashing batch jobs.
    • Itertools: Helper functions for iterating over data from unreliable sources.

    2. FastWARC

    A high-performance WARC parsing library for Python (written in Rust). It supports compressed and uncompressed WARC/1.0 and WARC/1.1 streams using GZip, Zstd, and LZ4 compression algorithms.

  7. What is FastWARC and its supported formats

    develop

    FastWARC is a high-performance Python library for parsing WARC files, built with C++/Cython. While its API is inspired by warcio, it is not a drop-in replacement.

    Supported Formats & Compression:

    • WARC Versions: WARC/1.0 and WARC/1.1 (both compressed and uncompressed).
    • Compression Algorithms: Gzip and LZ4.

    FastWARC can be used via its Python API or its Command Line Interface (CLI).

  8. What is the Resiliparse Process Guard and when to use it

    develop

    The Process Guard module provides decorators and context managers designed to protect batch processing jobs from individual task failures.

    Use Process Guards when running large-scale jobs where individual tasks might:

    • Exceed pre-defined execution time limits (timeouts).
    • Consume abnormal amounts of memory.

    By wrapping a processing context in a guard, the task will be interrupted if it exceeds resource limits, allowing the overall job to continue successfully by skipping or rescheduling the problematic task rather than crashing the entire process.

  9. What are Resiliparse Process Guards?

    develop

    Resiliparse Process Guards are a set of decorators and context managers designed to monitor and limit the execution time and memory usage of running tasks. They are useful for batch processing jobs where individual tasks might time out or consume excessive memory, allowing the overall job to continue by skipping or rescheduling failed tasks.

    Platform Support:

    • Linux: Full support for both TimeGuard and MemGuard.
    • macOS: TimeGuard is supported but not extensively tested. MemGuard is unsupported.
    • Windows: The Process Guards module is not available.

  10. Perform main content extraction (boilerplate removal)

    develop

    You can perform fast, rule-based main content extraction (boilerplate removal) by setting main_content=True. This applies rules to remove elements like navigation blocks, sidebars, footers, and ads.

    By default, the following are treated as main content:

    • News article or blog post meta information (author, date, etc.)
    • User comments

    To exclude these, set post_meta=False or comments=False. To include elements that are typically hidden via CSS or class names, set hidden_elements=True.

    # Basic main content extraction
print(extract_plain_text(html, main_content=True))

# Main content extraction with specific formatting overrides
print(extract_plain_text(html, 
                         main_content=True, 
                         alt_texts=False, 
                         preserve_formatting=False, 
                         noscript=True))

  11. Perform advanced DOM traversal with callbacks

    develop

    For complex traversal logic, use the traverse_dom function. This function accepts a DOMNode and two optional callback functions:

    1. start_cb: Called when the traverser encounters a node (start tag).
    2. end_cb: Called when the traverser encounters an element's end tag (going up one level).

    Each callback receives a DOMContext object, which provides:

    • node: The current DOMNode being visited.
    • depth: The current traversal depth.
    • value: The text content of the node (if it is a text node).
    • tag: The tag name (if it is an element node).

    Note: The DOMContext object is mutable. You can attach custom state to it (e.g., a stack) to maintain context across the traversal.

    from resiliparse.parse.html import *

def start_cb(ctx: DOMContext):
    if ctx.node.type == NodeType.ELEMENT:
        print(f'<{ctx.node.tag}>', end='')

def end_cb(ctx: DOMContext):
    if ctx.node.type == NodeType.ELEMENT:
        print(f'</{ctx.node.tag}>', end='')

traverse_dom(tree.body, start_cb, end_cb)

  12. How MatchFiles works in Resiliparse Beam

    develop

    The resiliparse.beam.fileio.MatchFiles transform is a replacement for Apache Beam's native MatchFiles. It forces a fusion break by adding a Reshuffle transform. This is specifically designed to fix shortcomings in the FlinkRunner, which currently does not support distributing splits on its own.

    If you are using the DataflowRunner or do not want to reshuffle matched file names, you can set the shuffle parameter to False or use the standard Beam implementation.