DOCSEARCHbyAlgolia
Submit Docs

Readability.js

repository·main·Indexed 11 days ago

https://github.com/mozilla/readability

A standalone implementation of the library used in Firefox Reader View. It extracts core content (title, text, images) from a DOM document while stripping clutter like ads and navigation. Version 0.6.0 provides the Readability class for parsing and the isProbablyReaderable function for suitability checks.

Tokens
2.5K
Snippets
9
Records
10
Agent score
45%

What's inside Readability

  1. Use Readability in Node.js with jsdom

    main

    Since Node.js lacks a built-in DOM, you must use an external library like jsdom to provide a document object.

    Important: When initializing JSDOM, pass the page's URI as the url option. This allows Readability to correctly convert relative URLs (for images, links, etc.) into absolute URLs.

    var { Readability } = require('@mozilla/readability');
var { JSDOM } = require('jsdom');
var doc = new JSDOM("<body>Look at this cat: <img src='./cat.jpg'></body>", {
  url: "https://www.example.com/the-page-i-got-the-source-from"
});
let reader = new Readability(doc.window.document);
let article = reader.parse();

  3. Security considerations for untrusted input

    main

    When using Readability with untrusted HTML or DOM input, the output may contain potentially malicious content.

    Recommendations:

    1. Use a sanitizer library like DOMPurify on the output of Readability to prevent script injection.
    2. Implement Content Security Policy (CSP) as a defense-in-depth measure.

    Readability does not sanitize unsafe content from the input automatically.

  4. Basic usage of Readability

    main

    To parse a document, create a new Readability instance from a DOM document object and call the .parse() method.

    Note: The .parse() method modifies the DOM. If you need to preserve the original document, pass a clone of the document to the constructor instead.

    // Standard usage
var article = new Readability(document).parse();

// Usage with a clone to avoid modifying the original DOM
var documentClone = document.cloneNode(true);
var article = new Readability(documentClone).parse();

  5. Check if a document is readable with isProbablyReaderable()

    main

    Use isProbablyReaderable(document, options) to perform a fast, low-cost check to see if a document is likely suitable for parsing. This is useful for avoiding expensive parsing logic on documents that are unlikely to yield a meaningful article. Note that it may produce false positives and false negatives.

    /*
    Only instantiate Readability  if we suspect
    the `parse()` method will produce a meaningful result.
*/
if (isProbablyReaderable(document)) {
    let article = new Readability(document).parse();
}

  6. Configure Readability with options

    main

    The new Readability(document, options) constructor accepts an optional options object to customize the parsing behavior.

    // Example of passing options
var article = new Readability(document, {
  debug: true,
  charThreshold: 1000,
  keepClasses: true
}).parse();

  7. Reference: isProbablyReaderable() options

    main

    The following properties can be passed in the options object to isProbablyReaderable(document, options):

    * `minContentLength` (number, default `140`): the minimum node content length used to decide if the document is readerable;
* `minScore` (number, default `20`): the minimum cumulated 'score' used to determine if the document is readerable;
* `visibilityChecker` (function, default `isNodeVisible`): the function used to determine if a node is visible;

  8. Reference: Readability constructor options

    main

    The following properties can be passed in the options object to new Readability(document, options):

    * `debug` (boolean, default `false`): whether to enable logging.
* `maxElemsToParse` (number, default `0` i.e. no limit): the maximum number of elements to parse.
* `nbTopCandidates` (number, default `5`): the number of top candidates to consider when analysing how tight the competition is among candidates.
* `charThreshold` (number, default `500`): the number of characters an article must have in order to return a result.
* `classesToPreserve` (array): a set of classes to preserve on HTML elements when the `keepClasses` options is set to `false`.
* `keepClasses` (boolean, default `false`): whether to preserve all classes on HTML elements. When set to `false` only classes specified in the `classesToPreserve` array are kept.
* `disableJSONLD` (boolean, default `false`): when extracting page metadata, Readability gives precedence to Schema.org fields specified in the JSON-LD format. Set this option to `true` to skip JSON-LD parsing.
* `serializer` (function, default `el => el.innerHTML`) controls how the `content` property returned by the `parse()` method is produced from the root DOM element. It may be useful to specify the `serializer` as the identity function (`el => el`) to obtain a DOM element instead of a string for `content` if you plan to process it further.
* `allowedVideoRegex` (RegExp, default `undefined` ): a regular expression that matches video URLs that should be allowed to be included in the article content. If `undefined`, the default regex is applied.
* `linkDensityModifier` (number, default `0`): a number that is added to the base link density threshold during the shadiness checks. This can be used to penalize nodes with a high link density or vice versa.

  9. Reference: parse() return object

    main

    The .parse() method returns an object containing the extracted article data:

    * `title`: article title;
* `content`: HTML string of processed article content;
* `textContent`: text content of the article, with all the HTML tags removed;
* `length`: length of an article, in characters;
* `excerpt`: article description, or short excerpt from the content;
* `byline`: author metadata;
* `dir`: content direction;
* `siteName`: name of the site;
* `lang`: content language;
* `publishedTime`: published time;

  10. Use the Readability API

    main

    The @mozilla/readability package provides two primary exports for extracting content from a DOM document: the Readability class for parsing and isProbablyReaderable for checking if a document is suitable for parsing.

    To use them in a Node.js environment, require the module and use the following patterns:

    Parsing a document

    Initialize a new Readability instance with a document object and optional options, then call the .parse() method.

    Checking readability

    Use isProbablyReaderable(document, options) to determine if a document contains enough content to be successfully processed by the parser.

    const { Readability, isProbablyReaderable } = require('@mozilla/readability');

// Check if document is suitable
if (isProbablyReaderable(doc)) {
  // Parse the document
  const reader = new Readability(doc);
  const article = reader.parse();
  console.log(article.textContent);
}