html5lib Documentation

The html5lib package provides tools for parsing and serializing HTML according to the HTML5 specification. The core functionality is divided into several modules:

• html5lib.html5parser: Used for parsing HTML content into a tree structure.
• html5lib.serializer: Used for converting tree structures back into HTML strings via the HTMLSerializer class.
• html5lib.constants: Contains various constants used throughout the library.

For advanced manipulation of the parsed tree, you can use the following subpackages:

• html5lib.filters: For filtering tree nodes.
• html5lib.treebuilders: For defining how the tree is constructed.
• html5lib.treewalkers: For traversing the tree.
• html5lib.treeadapters: For adapting different tree implementations to the library's interface.

html5lib uses a pipeline of components to process HTML:

1. Tree Builders: Parse tokenized content into an in-memory tree representation.
2. Tree Walkers: Translate a tree into a streaming token stream.
3. Filters: Transform the token stream (e.g., for sanitization or sorting).
4. HTMLSerializer: Converts the token stream back into a stream of bytes.
5. Tree Adapters: Translate trees between different formats (e.g., to SAX or Genshi).

The benchmarks/data/wpt directory contains a subset of tests from the web-platform-tests repository. These tests are used for benchmarking and are organized into two distinct categories to provide different coverage profiles:

• weighted: A curated set of 15 tests chosen via a random weighted sample (weighted by parse time as of html5lib 1.0.1). This set focuses on files that significantly contribute to manifest generation time, effectively targeting the slowest files.
• random: A set of 15 tests chosen via a random unweighted sample. This set provides a profile closer to the average file found in the WPT repository.

The html5lib.treebuilders package provides different implementations for building the tree structure from parsed HTML. Depending on your needs for performance or compatibility with other libraries, you can choose between several modules:

• dom: Builds a standard DOM tree.
• etree: Builds a tree using the standard Python xml.etree.ElementTree API.
• etree_lxml: Builds a tree using the lxml library (requires lxml to be installed), which is typically faster and more feature-rich than the standard etree.

To use a specific treebuilder, you pass the module or a class from these modules to the html5lib.parse function (or the relevant parser constructor).

You can choose how the in-memory tree is constructed. html5lib supports three types:

• etree: The default. Uses xml.etree.ElementTree (or the accelerated cElementTree).
• dom: Uses xml.dom.minidom.
• lxml: Uses lxml.etree (provides an ElementTree API).

Using the shorthand API: Pass the treebuilder argument to html5lib.parse().

Using the HTMLParser class: Pass a builder class to the tree keyword argument when instantiating html5lib.HTMLParser.

import html5lib
# Shorthand API
with open("mydocument.html", "rb") as f:
    lxml_etree_document = html5lib.parse(f, treebuilder="lxml")

# Using HTMLParser class
TreeBuilder = html5lib.getTreeBuilder("dom")
parser = html5lib.HTMLParser(tree=TreeBuilder)
minidom_document = parser.parse("<p>Hello World!")

Parsed trees are always Unicode. To handle various input encodings, html5lib follows this priority:

1. Explicit Specification: Pass the encoding name to the encoding parameter of html5lib.html5parser.HTMLParser.parse().
2. Meta Element: If not specified, the parser attempts to detect encoding from a <meta> element in the first 512 bytes.
3. Sniffing: If no meta element is found and the chardet library is installed, it attempts to sniff the encoding from the byte pattern.
4. Fallback: If all else fails, it defaults to Windows-1252.

Filters can be used to transform a token stream. To use a filter, wrap it around a stream generated by a tree walker.

Available filters include:

• alphabeticalattributes.Filter: Sorts attributes on tags alphabetically.
• inject_meta_charset.Filter: Sets a user-specified encoding in the <meta> tag.
• lint.Filter: Raises AssertionError on invalid names or PCDATA.
• optionaltags.Filter: Removes unnecessary tags.
• sanitizer.Filter: Removes unsafe markup and CSS (follows WHATWG rules).
• whitespace.Filter: Collapses whitespace (except in <pre> or <textarea>).

import html5lib
from html5lib.filters import sanitizer

dom = html5lib.parse("<p><script>alert('Boo!')", treebuilder="dom")
walker = html5lib.getTreeWalker("dom")
stream = walker(dom)
clean_stream = sanitizer.Filter(stream)

html5lib works on CPython 2.7+, CPython 3.5+, and PyPy. You can install it using pip:

$ pip install html5lib

When parsing content fetched via HTTP, you should pass the detected charset to html5lib using the transport_encoding argument to ensure correct parsing.

# Python 3 usage with urllib.request
from urllib.request import urlopen
import html5lib

with urlopen("http://example.com/") as f:
    document = html5lib.parse(f, transport_encoding=f.info().get_content_charset())

# Python 2 usage with urllib2
from contextlib import closing
from urllib2 import urlopen
import html5lib

with closing(urlopen("http://example.com/")) as f:
    document = html5lib.parse(f, transport_encoding=f.info().getparam("charset"))

To write HTML back as a stream of bytes, use html5lib.serializer.HTMLSerializer. The typical workflow is:

1. Parse the document into a tree.
2. Use a Tree Walker to create a token stream from that tree.
3. Pass the stream to the serializer.

import html5lib
# 1. Parse
element = html5lib.parse('<p xml:lang="pl">Witam wszystkich')
# 2. Walk
walker = html5lib.getTreeWalker("etree")
stream = walker(element)
# 3. Serialize
s = html5lib.serializer.HTMLSerializer()
output = s.serialize(stream)
for item in output:
    print("%r" % item)

For more granular control, instantiate an html5lib.HTMLParser object explicitly. For example, you can enable strict mode to make the parser raise exceptions on parse errors.

import html5lib

# Create a strict parser
with open("mydocument.html", "rb") as f:
    parser = html5lib.HTMLParser(strict=True)
    document = parser.parse(f)

# Create a parser with a specific treebuilder class
parser = html5lib.HTMLParser(tree=html5lib.getTreeBuilder("dom"))
minidom_document = parser.parse("<p>Hello World!")