DOCSEARCHbyAlgolia
Submit Docs

trafilatura

repository·master·Indexed 27 days ago

https://github.com/adbar/trafilatura

A high-performance Python library and command-line tool for web crawling and extracting structured text and metadata from HTML. It supports sitemaps, feeds, and parallel processing to gather content while filtering out noise like headers and footers. Trafilatura provides multiple output formats including TXT, Markdown, CSV, JSON, HTML, XML, and XML-TEI, and includes core functions such as extract(), extract_with_metadata(), and focused_crawler().

Tokens
30.4K
Snippets
76
Records
232
Agent score
92%

What's inside trafilatura

  1. Overview of Trafilatura

    master
    Trafilatura is a Python package and command-line tool designed for web crawling, downloading, scraping, and extracting structured text data from raw HTML. It focuses on extracting main text, metadata, and comments while filtering out noise like headers and footers. It is modular, requires no database, and supports multiple output formats.

  2. Understand the Trafilatura role in corpus construction

    master

    Trafilatura is designed to handle the primary technical phases of building a web corpus: Web Crawling (determining range and content) and Data Pre-processing (cleaning and de-duplicating).

    In a typical web corpus workflow, Trafilatura addresses the 'Web Scraping and Data Cleaning' stage, which involves:

    • Removing boilerplate (navigation bars, headers, footers, etc.).
    • Identifying metadata, paragraphs, and structural information.
    • Producing output in a standard form suitable for linguistic analysis.
    • Stripping unnecessary content to obtain clean documents.

  3. Note on Graphical User Interface (GUI) availability

    master

    The Graphical User Interface (GUI) has been removed as of Trafilatura version 2.0 due to maintenance and cross-platform testing issues.

    If you are using a version of Trafilatura $\ge$ 2.0, you should use the Command Line Interface (CLI) instead. The GUI was only available in versions up to 1.8.1.

  4. Trafilatura Features

    master

    Trafilatura provides several key capabilities for web data extraction:

    • Web Crawling & Discovery: Supports sitemaps (TXT, XML) and feeds (ATOM, JSON, RSS) with smart URL management (filtering and deduplication).
    • Parallel Processing: Handles live URLs and previously downloaded HTML files/parsed trees efficiently.
    • Robust Extraction: Extracts main text (using algorithms like jusText and readability), metadata (title, author, date, site name, categories, tags), and optional elements (comments, links, images, tables).
    • Multiple Output Formats: Supports TXT, Markdown, CSV, JSON, HTML, XML, and XML-TEI.
    • Add-ons: Includes language detection and speed optimizations.

  5. Quickstart with Trafilatura CLI

    master

    You can use the Trafilatura CLI to extract text from URLs directly or by piping HTML content into it.

    Extract from a URL: Use the -u or --URL flag to provide a web address.

    Extract from piped content: You can pipe the contents of a local file or the output of a download utility (like wget or curl) directly into trafilatura via STDIN.

  6. Install Trafilatura with all optional dependencies

    master

    Trafilatura supports optional modules for extended functionality like faster encoding detection, language detection, and improved compression. You can install all these additional features at once using the [all] extra specifier.

    $ pip install trafilatura[all]

  7. Access the Trafilatura API

    master

    The Trafilatura API allows you to download URLs or provide your own data for text extraction with configurable output formats. It is useful for testing the software without local installation.

    Note: The API is currently listed as unavailable. Check the following gateways for status or inquiries:

    • Free demonstration gateway: https://trafilatura.mooo.com
    • High-volume gateway (via RapidAPI): https://rapidapi.com/trafapi/api/trafilatura

  9. Extract links from social media posts

    master

    To extract URLs from social media content like tweets, use a regular expression to find web links.

    Warning: Extracted links may be shortened (e.g., t.co/...) and must be resolved to their actual targets before processing with Trafilatura.

    re.findall(r'https?://[^ ]+')

  12. Install and update Trafilatura

    master

    To use Trafilatura, install it via pip. To ensure you have the latest version, use the -U flag.

    For advanced tokenization tasks (splitting text into words and sentences), it is recommended to also install SoMaJo.

    pip install -U trafilatura
pip install -U SoMaJo