DOCSEARCHbyAlgolia
Submit Docs

html5ever Documentation

repository·main·Indexed 25 days ago

https://github.com/servo/html5ever

A high-performance, browser-grade HTML5 parser written in Rust and designed to follow WHATWG specifications. Part of the Servo browser engine, it focuses on providing hooks for production-grade web browser behavior. The project includes related crates such as xml5ever for XML parsing, tendril for zero-copy string/buffer optimization, and markup5ever_rcdom for automated testing.

Tokens
10.5K
Snippets
22
Records
92
Agent score
79%

What's inside html5ever

  1. Overview of html5ever

    main

    html5ever is an HTML parser developed as part of the Servo project. It parses and serializes HTML according to the WHATWG (HTML5) specifications.

    Key characteristics:

    • Performance & Safety: Written in Rust to provide C-like performance without the security risks of C or the need for a garbage collector.
    • DOM Manipulation: Unlike many parsers, html5ever does not provide a built-in DOM tree representation; instead, it uses callbacks to manipulate the DOM.
    • Encoding: Exclusively uses UTF-8 for string representation.
    • Compatibility: Passes tokenizer tests from html5lib-tests and aims to provide hooks necessary for production web browsers (e.g., document.write).

    Note: For XHTML, it is recommended to use an XML parser, as html5ever is optimized for HTML5.

  2. Introduction to Tendril

    main

    Overview

    Tendril is a compact string/buffer type optimized for zero-copy parsing. It provides owned-string semantics but can act as a view into shared buffers.

    Key Characteristics

    • Zero-copy optimization: Uses thread-local (non-atomic) reference counting for low-overhead buffer sharing.
    • Memory Efficiency:
      • Stores small strings (up to 8 bytes) in-line without heap allocation.
      • Smaller footprint than String on 64-bit platforms (16 bytes vs 24 bytes).
      • Option<Tendril> is the same size as Tendril due to NonZero optimization.
    • Constraints:
      • Maximum length is 4 GB (exceeding this will cause a panic).
      • Thread Safety: The Rust type system prevents sending a Tendril between threads.

    Warning: This library is at a very early stage of development and contains a substantial amount of unsafe code. Use at your own risk.

  3. Use markup5ever_rcdom for automated testing

    main

    The markup5ever_rcdom crate is designed specifically for writing automated tests for the html5ever and xml5ever crates.

    Warning: This is not a production-quality DOM implementation. It has not been fuzzed or tested against arbitrary, malicious, or nontrivial inputs. Use in production or user-facing systems is at your own risk, as no maintenance or support is provided for such use cases.

  8. Implement a TokenSink to process XML tokens

    main

    To process a stream of XML tokens (like start tags, end tags, or comments), implement the TokenSink trait. You define a struct and implement the process_token method, which receives a Token enum. This allows you to react to different parts of the XML stream as they are parsed.

    Dependencies

    Add these to your Cargo.toml:

    [dependencies]
xml5ever = "0.2.0"
tendril = "0.1.3"

    Implementation Pattern

    1. Define a struct to act as your sink.
    2. Implement TokenSink for that struct.
    3. Use XmlTokenizer to feed input into your sink.
    struct SimpleTokenPrinter;

impl TokenSink for SimpleTokenPrinter {
    fn process_token(&mut self, token: Token) {
        match token {
            CharacterTokens(b) => {
                println!("TEXT: {}", &*b);
            },
            NullCharacterToken => print!("NULL"),
            TagToken(tag) => {
                println!("{:?} {} ", tag.kind, &*tag.name.local);
            },
            ParseError(err) => {
                println!("ERROR: {}", err);
            },
            PIToken(Pi{ref target, ref data}) => {
                println!("PI : <?{} {}?>", &*target, &*data);
            },
            CommentToken(ref comment) => {
                println!("<!--{:?}-->", &*comment);
            },
            EOFToken => {
                println!("EOF");
            },
            DoctypeToken(Doctype{ref name, ref public_id, ..}) => {
                println!("<!DOCTYPE {:?} {:?}>", &*name, &*public_id);
            }
        }
    }
}

fn main() {
    let sink = SimpleTokenPrinter;
    let mut input = ByteTendril::new();
    io::stdin().read_to_tendril(&mut input).unwrap();
    let input = input.try_reinterpret().unwrap();

    let mut tok = XmlTokenizer::new(sink, Default::default());
    tok.feed(input);
    tok.end(); // Must be called to process final bytes
}

  9. Parse XML into a DOM tree using TreeSink

    main

    To build a structured XML document tree instead of just processing individual tokens, use a TreeSink. The xml5ever crate provides a built-in implementation called RcDom.

    Usage

    1. Prepare your input as a StrTendril.
    2. Use the parse function combined with one_input to convert the input into an RcDom instance.
    3. Traverse the resulting tree by accessing the nodes and their children.

  10. Build and test xml5ever

    main

    To build the project and run tests (which may include tests fetched from external submodules like xml5lib-tests), follow these steps:

    1. Initialize submodules to fetch external tests.
    2. Build the crate.
    3. Run the tests.

    You can also generate local documentation using cargo docs or view the hosted API documentation.

    git submodule update --init # to fetch xml5lib-tests
cargo build
cargo test

  12. Use Tendril for compact string management

    main

    A Tendril is a compact string type designed for zero-copy parsing. It behaves like an owned string but can act as a view into shared buffers.

    Key characteristics:

    • Small String Optimization: Strings up to 8 bytes are stored inline without heap allocation.
    • Efficiency: On 64-bit platforms, a Tendril is 16 bytes (compared to 24 bytes for String).
    • Memory Semantics: Mutating a Tendril that is currently a view into a shared buffer will trigger an owned copy (copy-on-write).
    • Thread Safety: By default, Tendril uses NonAtomic reference counting and is not Send. To make it thread-safe, use the Atomic type parameter.