Readability4J

Readability4J is designed to be extensible. You can customize the parsing logic by providing your own implementations of the core processing classes to the Readability4J constructor. The following classes can be overwritten:

• Preprocessor (handles _removeScripts() and _prepDocument())
• ArticleGrabber (handles _grabArticle())
• Postprocessor (handles _postProcessContent())
• MetadataParser (handles _getArticleMetadata())

To extract the main content from a webpage, instantiate Readability4J with the page URL and the HTML content. The URL is required to resolve relative URLs within the document. Call .parse() to obtain an Article object containing the extracted data.

String url = ...;
String html = ...;

Readability4J readability4J = new Readability4J(url, html); // url is just needed to resolve relative urls
Article article = readability4J.parse();

// Accessing extracted data:
String extractedContentHtml = article.getContent();
String extractedContentPlainText = article.getTextContent();
String title = article.getTitle();
String byline = article.getByline();
String excerpt = article.getExcerpt();

To use Readability4J in your project, add the following dependency to your build configuration.

dependencies {
  compile 'net.dankito.readability4j:readability4j:1.0.8'
}

<dependency>
   <groupId>net.dankito.readability4j</groupId>
   <artifactId>readability4j</artifactId>
   <version>1.0.8</version>
</dependency>

If you need features not present in the standard Mozilla Readability implementation, use Readability4JExtended. This version includes improvements such as:

• Copying URLs from data-src attributes to <img> src attributes to support lazy-loaded images.
• Using <head><base>'s href value for resolving relative URLs.
• Improved detection of which images to keep in the output.

Usage is identical to the standard class, just swap the constructor.

Readability4J readability4J = new Readability4JExtended(url, html);
Article article = readability4J.parse();

By default, article.getContent() returns a <div> element without an encoding declaration, which can cause issues with non-ASCII characters. To ensure correct display, use the convenience methods in the Article class that wrap the content in a full <html> structure with a <meta charset> tag.

// Wraps content in <html><head><meta charset="utf-8" /></head><body>...</body></html>
String contentHtmlWithUtf8Encoding = article.getContentWithUtf8Encoding();

// Tries to apply site's charset, or uses UTF-8 as fallback
String contentWithDocumentsCharsetOrUtf8 = article.getContentWithDocumentsCharsetOrUtf8();

// Wraps content with a specific custom encoding
String contentHtmlWithCustomEncoding = article.getContentWithEncoding("ISO-8859-1");