unfluff

You can install unfluff either as a global command-line utility or as a Node.js module for programmatic use.

To install the CLI utility globally:

npm install -g unfluff

To install the module for use in your Node.js project:

npm install --save unfluff

The extractor(html, language) function is the primary way to parse HTML content programmatically. It returns a JSON object containing all extracted elements.

Parameters:

• html (String): The HTML content you want to parse.
• language (String, optional): A two-letter language code. If not provided, the library attempts to auto-detect the language. Note that the extraction algorithm depends heavily on the language setting.

Example:

const extractor = require('unfluff');
const data = extractor(my_html_data, 'en');
console.log(data.text);

If you are processing large documents and only need a few specific elements (like title or image), use extractor.lazy(html, language). This avoids the overhead of the full text extraction pipeline.

Unlike the standard extractor, which returns a JSON object, extractor.lazy returns an object where every field is a function. Evaluation is only performed when the function is called, and results are cached for subsequent calls.

Example:

const extractor = require('unfluff');
const data = extractor.lazy(my_html_data, 'en');

// Access elements by calling them as functions
console.log(data.title());
console.log(data.image());
console.log(data.text());

The unfluff command-line interface allows you to parse web pages by passing a filename or piping HTML content into it. The output is a JSON object containing the extracted data.

Pass a file:

unfluff my_file.html

Pipe content (e.g., via curl):

curl -s "http://somesite.com/page" | unfluff

Example: Extract only the body text using jq:

curl -s "https://www.polygon.com/2014/6/26/5842180/shovel-knight-review-pc-3ds-wii-u" | unfluff | jq -r .text

To extract the core content of an article, use the text and links methods. These methods rely on a topNode (the container element identified as the main content) to avoid extracting noise from sidebars or footers.

• text(doc, topNode, lang): Returns the cleaned, formatted text of the main article. It uses a scoring algorithm to find the best content node and performs post-extraction cleanup.
• links(doc, topNode, lang): Returns an array of objects containing { text, href } for all links found within the topNode.
• videos(doc, topNode): Scans the topNode for video elements (iframe, embed, object, video) and returns an array of video objects containing src, height, and width.

The calculateBestNode(doc, lang) function implements a scoring algorithm to identify the most likely container for the main article text.

It works by:

1. Scanning p, pre, and td elements.
2. Calculating a 'gravity score' based on stopword density (using the provided lang) and link density.
3. Boosting nodes that appear to be part of a continuous text flow.
4. Penalizing nodes with high link density or those located at the very bottom of the document.
5. Returning the DOM node with the highest accumulated score.

This node should be passed as the topNode to text(), links(), and videos() to ensure high-quality extraction.

If you only need specific parts of a document (e.g., just the title or just the text), use unfluff.lazy(html, language). This returns an object of getter functions. Each function, when called, extracts and caches its specific value. This avoids the overhead of cleaning the document and calculating the best content node unless those specific properties are requested.

Parameters:

• html (String): The raw HTML content.
• language (String, optional): The language of the document.

Returned Object Methods:

Metadata Getters (Parsed from raw HTML):

• title()
• softTitle()
• date()
• copyright()
• author()
• publisher()
• favicon()
• description()
• keywords()
• lang()
• canonicalLink()
• tags()
• image()

Content Getters (Require document cleaning and node calculation):

• videos()
• text()
• links()

The following options are available when using the unfluff command line interface: