DOCSEARCHbyAlgolia
Submit Docs

html5gum

repository·main·Indexed 41 days ago

https://github.com/untitaker/html5gum

A high-performance, WHATWG-compliant HTML5 tokenizer and tag soup parser written in Rust. It provides multiple API patterns including an Iterator API, a custom Emitter trait for maximum performance, a callback-based API via CallbackEmitter, and an experimental tree-builder feature for integration with scraper and html5ever.

Tokens
10.3K
Snippets
33
Records
47
Agent score
50%

What's inside html5gum

  1. Choose an API pattern for html5gum

    main

    Depending on your performance and convenience requirements, html5gum provides four primary ways to consume tokens:

    1. Iterator API: Iterate over tokens using Tokenizer::new(html). Best for general purpose use and ease of implementation.
    2. Custom Emitter API: Implement your own Emitter trait for maximum performance. This is the fastest way to process or transform HTML.
    3. Callback-based API: A middleground between convenience and performance using callbacks.
    4. Tree-builder API: By enabling the tree-builder feature, you can integrate html5gum with html5ever and scraper to perform full DOM-based parsing.

  2. Understand the limitations of the tokenizer

    main

    Because html5gum is a WHATWG-compliant tokenizer and not a full browser-grade parser, it has specific behaviors you must account for:

    • No Charset Detection: It assumes UTF-8 and operates on bytes. It does not implement the charset detection phase of the HTML spec.
    • No Tag Correction: It does not correct mis-nested tags; it simply tokenizes the input as provided.
    • No DOM Implementation: It does not build a Document Object Model (DOM). In the HTML spec, tree construction can influence tokenization; since html5gum tokenizes in isolation, it may behave differently than a browser in specific edge cases involving state switches.
    • Byte-based: The implementation takes and returns bytes.

  3. Manage the fuzzing corpus

    main

    The in/ directory contains test files used to seed the fuzzer. These are initially populated from the html5lib-tests submodule.

    • Initialize corpus: Run make in to explicitly populate the directory.
    • Custom seeds: You can manually modify the contents of in/ to add your own test cases.
    • Interoperability: cargo-fuzz and AFL can be run in alternating ways. cargo-fuzz reads from both in/ and out/, while AFL reads from in/ and manages its queue in out/.
    make in

  4. Iterate over HTML tokens with Tokenizer

    main

    The simplest way to use html5gum is to create a new Tokenizer from an HTML string (or byte slice) and iterate over the resulting tokens. Each token is returned as a Token enum variant. Note that the tokenizer works with bytes and assumes UTF-8, though it recovers gracefully from invalid UTF-8 sequences.

    use std::fmt::Write;
use html5gum::{Tokenizer, Token};

let html = "<title   >hello world</title>";
let mut new_html = String::new();

for Ok(token) in Tokenizer::new(html) {
    match token {
        Token::StartTag(tag) => {
            write!(new_html, "<{}>", String::from_utf8_lossy(&tag.name)).unwrap();
        }
        Token::String(hello_world) => {
            write!(new_html, "{}", String::from_utf8_lossy(&hello_world)).unwrap();
        }
        Token::EndTag(tag) => {
            write!(new_html, "</{}>", String::from_utf8_lossy(&tag.name)).unwrap();
        }
        _ => panic!("unexpected input"),
    }
}

assert_eq!(new_html, "<title>hello world</title>");

  5. Run fuzzing with AFL

    main

    AFL is the more mature fuzzing setup for html5gum. Use the following commands via make:

    • Basic fuzz (crashes only): FUZZ_BASIC=1 make -e afl
    • Multi-core fuzzing (Slave fuzzer): FUZZ_BASIC=1 _AFL_OPTS='-S fuzzer02' make -e afl (The master is fuzzer01).
    • Process next crash: FUZZ_BASIC=1 make -e afl-next (Runs afl-tmin on the next crash and prints the testcase as a JSON string).
    • Skip result: FUZZ_BASIC=1 make -e afl-skip.
    # Basic AFL fuzz
FUZZ_BASIC=1 make -e afl

# Run a slave fuzzer for multi-core
FUZZ_BASIC=1 _AFL_OPTS='-S fuzzer02' make -e afl

  6. Integrate with scraper and html5ever via tree-builder

    main
    If you need to perform DOM-based querying (e.g., using CSS selectors), you can use the experimental tree-builder feature. This allows html5gum to work with the scraper and html5ever ecosystem, effectively turning the tokenizer into a full parser.

  7. Run fuzzing with cargo-fuzz

    main

    To use cargo-fuzz for testing html5gum:

    • Basic fuzz: FUZZ_BASIC=1 make -e libfuzzer
    • Multi-core fuzzing: Use the _LIBFUZZER_OPTS environment variable. Example: _LIBFUZZER_OPTS='--jobs 8'.
    • Interactive shell: Run FUZZ_BASIC=1 make -e sh to enter a shell where all fuzz target environment variables are pre-set. From there, you can use standard cargo-fuzz subcommands to manage testcases.

    Note: By default, no sanitizer is run because the default ASAN causes too many false-positives.

    # Basic libfuzzer run
FUZZ_BASIC=1 make -e libfuzzer

# Multi-core libfuzzer run
FUZZ_BASIC=1 _LIBFUZZER_OPTS='--jobs 8' make -e libfuzzer

  8. Structure of CharRef in HTML entity parsing

    main

    When parsing HTML character references, the library uses a CharRef structure to represent the mapping between an entity name and its corresponding character(s). Each CharRef contains:

    • name: The string identifier used in the HTML entity (e.g., "ildeFullEqual;").
    • characters: The actual Unicode character(s) the entity represents (e.g., "\u{2245}").

  9. Structure of CharRef for HTML entities

    main

    In html5gum, HTML character references (entities) are represented by the CharRef struct. When a character reference is parsed, it maps a specific entity name to its corresponding Unicode characters.

    Each CharRef contains:

    • name: The string identifier of the entity (e.g., "quot", "amp", or "copy").
    • characters: The actual Unicode string that the entity represents.

  10. Track byte ranges using Span and Spanned

    main

    The Spanned<T, B> type allows you to associate a value of type T with its location in the input source using a Span<B>.

    By default, the bound type B is usize, representing byte offsets. However, you can use other types that implement the SpanBound trait. If you want to ignore spans entirely, you can use the unit type () as the bound type.

    Spanned implements Deref and DerefMut, so you can access the inner value directly as if it were the underlying type.

    use html5gum::Spanned;

// Using the default usize bound
let spanned: Spanned<String, usize> = "hello".to_string().into();
assert_eq!(spanned.len(), 5); // Accesses String::len() via Deref

// Using () to ignore spans
let ignored: Spanned<String, ()> = "hello".to_string().into();

  11. Use ForwardingEmitter to wrap another emitter

    main
    If you want to create an emitter that mostly wraps and forwards calls to another existing emitter, implement the ForwardingEmitter trait. This is more efficient than implementing the full Emitter trait from scratch when you only need to intercept or modify a few specific methods.

  12. Initialize a Tokenizer

    main

    You can create a new Tokenizer using Tokenizer::new(input) if you want to use the DefaultEmitter. The input must implement the Readable trait (e.g., &str or String).

    If you need more control over string allocation or want to use a custom Emitter, use Tokenizer::new_with_emitter(input, emitter).

    use html5gum::Tokenizer;

// Using the default emitter with a string slice
let tokenizer = Tokenizer::new("<html><body></body></html>");