jsoup Java Library

jsoup is a Java library designed for working with real-world HTML and XML. It implements the WHATWG HTML5 specification, ensuring it parses HTML into a DOM similar to modern web browsers.

Key capabilities include:

• Scraping and Parsing: Fetch and parse HTML from URLs, files, or strings.
• Data Extraction: Find and extract data using DOM traversal or CSS selectors.
• Manipulation: Modify HTML elements, attributes, and text.
• Sanitization: Clean user-submitted content against a safe-list to prevent XSS attacks.
• Output: Generate tidy HTML output.

To use jsoup in your project:

1. Download the latest jsoup JAR or add it as a dependency to your Maven or Gradle build.
2. Consult the jsoup cookbook for specific implementation patterns.

You can use Jsoup.connect(url).get() to fetch a web page and parse it into a Document object. Once parsed, you can use CSS selectors via the .select() method to find specific Elements.

Document doc = Jsoup.connect("https://en.wikipedia.org/").get();
log(doc.title());
Elements newsHeadlines = doc.select("#mp-itn b a");
for (Element headline : newsHeadlines) {
  log("%s\n\t%s", 
    headline.attr("title"), headline.absUrl("href"));
}

By default, jsoup uses the Java 11 HttpClient to execute requests, which provides support for HTTP/2. If you need to disable this behavior and fallback to the standard UrlConnection implementation, set the following system property to false:

jsoup.useHttpClient=false

System.setProperty("jsoup.useHttpClient", "false");

By default, jsoup attempts to use HttpClient if it is available on the classpath. If you need to explicitly prefer the standard HttpURLConnection implementation (for example, to avoid dependencies or for troubleshooting connection behavior), set the following system property to false:

jsoup.useHttpClient=false

System.setProperty("jsoup.useHttpClient", "false");

jsoup uses an internal AuthenticationHandler to manage credentials during HTTP requests. While the handler itself is package-private, the mechanism relies on providing a RequestAuthenticator implementation. This allows you to define custom logic to provide PasswordAuthentication based on the request context (URL, requestor type, and prompt).

When running on Java 9+, jsoup attempts to use a per-request RequestAuthHandler. On Java 8, it uses a system-wide Authenticator that delegates to a ThreadLocal pool to ensure thread safety for per-request credentials.