DOCSEARCHbyAlgolia
Submit Docs

htmldate

repository·master·Indexed 41 days ago

https://github.com/adbar/htmldate

A Python tool for fast and robust extraction of original and updated publication dates from URLs and web pages. It uses a multi-layered heuristic approach analyzing HTML headers, structural markers, and bare content. Features include a CLI for single or batch processing, a Python API via the find_date function, and optional linguistic analysis for improved accuracy through the [all] installation extra.

Tokens
8K
Snippets
35
Records
47
Agent score
96%

What's inside htmldate

  1. How htmldate extracts dates

    master

    Htmldate uses a multi-layered heuristic approach to identify dates within HTML markup and text:

    1. Markup in header: Searches for link and meta elements, including Open Graph protocol attributes.
    2. HTML code: Scans the document for structural markers like abbr or time elements and specific attributes (e.g., postmetadata).
    3. Bare HTML content:
      • Fast mode: Cleans the HTML and targets precise patterns.
      • Extensive mode: Collects all potential dates and uses a disambiguation algorithm to select the most likely candidate.

    Finally, the detected date is validated and converted to the requested format (defaulting to ISO 8601 YMD).

  2. Understand the difference between fast and extensive modes in htmldate[all]

    master

    When using the htmldate[all] installation, the library can operate in different modes that balance speed and accuracy:

    • Fast mode: Prioritizes speed and precision. It outputs fewer dates (lower recall), but the dates it does output are more likely to be correct (higher precision).
    • Extensive mode: Prioritizes coverage. It uses more intensive linguistic analysis to find more dates (higher recall), which can lead to higher overall accuracy and F-Score, but it is slower than the fast mode.

  3. How htmldate extracts publication dates

    master

    htmldate finds original and updated publication dates of web pages using a combination of heuristics on HTML code and linguistic patterns. It accepts URLs, HTML files, or HTML trees as input and returns a date string (defaulting to ISO 8601 YMD format) or None if no valid date is found.

    The extraction process follows three main stages:

    1. Markup in header: Identifies relevant elements like <link> and <meta> tags, including Open Graph protocol attributes and CMS-specific idiosyncrasies.
    2. HTML code: Searches the document for structural markers such as <abbr> and <time> elements, as well as specific attributes like postmetadata.
    3. Bare HTML content: Runs heuristics on text and markup using two distinct modes:
      • fast mode: Cleans the HTML page and targets precise patterns for high precision.
      • extensive mode: Collects all potential dates and uses a disambiguation algorithm to determine the best one for higher recall.

    You can switch between retrieving the original publishing statement (default) or the last updated date.

  4. Reproduce htmldate evaluation benchmarks

    master

    To reproduce the performance evaluations documented in this project, follow these steps:

    1. Clone the repository.
    2. Install all necessary packages (including htmldate[all]).
    3. Run the evaluation script located at tests/comparison.py using the data provided in the tests directory.
    # Example workflow (requires repository access)
python tests/comparison.py

  5. Install htmldate

    master

    You can install htmldate using pip. It is compatible with Python 3.10 and later. For optimized performance, you can install the [speed] extra.

    Note on Python versions:

    • For Python 3.10+: Use the latest version.
    • For Python 3.8 and 3.9: Use the 1.9.x series.
    • For Python 3.6 and 3.7: Use htmldate==1.8.1.
    # Standard installation
pip install htmldate

# Installation with speed optimizations
pip install htmldate[speed]

  6. Install htmldate with extended features

    master

    By default, htmldate provides a fast mode. For improved accuracy and coverage through additional linguistic analysis, you can install the package with the [all] extra. This is recommended if you need higher recall and better date parsing capabilities.

    Use one of the following commands depending on your package manager:

    pip install htmldate[all]
pip3 install htmldate[all]
pipenv install htmldate[all]

  7. Install htmldate with all dependencies

    master

    To improve performance and coverage (including linguistic analysis for better accuracy and faster date parsing), you can install htmldate with its optional dependencies using the [all] extra. This is recommended for better results in non-English languages and more robust extraction.

    Use one of the following commands depending on your package manager:

    pip install htmldate[all]
# or
pip3 install htmldate[all]
# or
pipenv install htmldate[all]

  8. Use the htmldate CLI to find publication dates

    master

    The htmldate command-line interface allows you to extract publication dates from web pages using several input methods:

    1. Single URL: Provide a URL directly using the --URL flag.
    2. STDIN: Pipe HTML content directly into the command.
    3. Batch Processing: Provide a file containing a list of URLs using the --inputfile flag. The tool will process each URL and output the results in a tab-separated format: URL\tDATE.

    Input Modes Summary

    ModeCommand/FlagBehavior
    URL-u or --URL <url>Fetches the content of the provided URL and extracts the date.
    STDIN(No flag)Reads HTML content from standard input.
    Batch-i or --inputfile <file>Reads a file line-by-line, treating each line as a URL.

    Example Usage

    Extract date from a single URL:

    htmldate --URL https://example.com

    Batch process URLs from a file:

    htmldate --inputfile urls.txt

    Pipe HTML content via STDIN:

    curl -s https://example.com | htmldate
    # Single URL
htmldate --URL https://example.com

# Batch processing
htmldate --inputfile urls.txt

# STDIN
curl -s https://example.com | htmldate

  9. Configure date extraction behavior via find_date parameters

    master

    When using find_date, you can control the precision and type of date returned using these key arguments:

    • Original vs. Modified: Use original_date=True to find when an article was first published. Use original_date=False (default) to find the most recent update.
    • Search Depth: Use extensive_search=True for a thorough search of the page text and patterns, or extensive_search=False for a faster, more targeted search.
    • Date Constraints: Use min_date and max_date to filter out implausible dates (e.g., dates in the future or dates before the website existed).
    • Output Format: The returned string follows the outputformat provided, which accepts any standard Python datetime.strftime tokens.

  11. Provide different input formats to find_date

    master

    The find_date function is flexible and accepts several types of input to locate a publication date:

    1. URLs: Pass a string containing the URL directly. htmldate will detect it is a URL and attempt to fetch the content.
    2. HTML Strings: Pass the raw HTML content as a string.
    3. LXML Trees: Pass a pre-parsed lxml.html tree object.
    4. Requests Objects: You can manually fetch content using the requests library and pass the text to find_date.
    from htmldate import find_date
from lxml import html
import requests
from htmldate.utils import fetch_url

# 1. Using a URL string
find_date('https://example.com/article')

# 2. Using an LXML tree
mytree = html.fromstring('<html><body><span class="entry-date">July 12th, 2016</span></body></html>')
find_date(mytree)

# 3. Using requests
r = requests.get('https://creativecommons.org/about/')
find_date(r.text)

# 4. Using htmldate's own fetch_url
htmldoc = fetch_url('https://blog.wikimedia.org/2018/06/28/interactive-maps-now-in-your-language/')
find_date(htmldoc)

  12. Extract publication dates with find_date

    master

    The primary entry point for finding the publication date of a web page is htmldate.core.find_date. This function orchestrates the extraction process by examining headers, searching the page content, and parsing potential date strings found within the HTML.

    import htmldate

date = htmldate.core.find_date('https://example.com/article')