lol-html
repository·main·Indexed 24 days ago
https://github.com/cloudflare/lol-htmlA low-latency, streaming HTML rewriter and parser with a CSS selector-based API. Designed for minimal memory overhead and buffering, it allows developers to modify HTML content on the fly. It is available as a Rust library, with a JavaScript API via WebAssembly, and official C bindings, as well as unofficial bindings for Lua, Go, and Ruby.
What's inside lol-html
- LOL HTML (Low Output Latency streaming HTML) is a streaming HTML rewriter and parser that uses a CSS-selector based API. It is designed to modify HTML on the fly with minimal buffering, making it suitable for very large documents and environments with limited memory resources. It serves as the backend for HTML rewriting in Cloudflare Workers but is available as a standalone library.
Run benchmarks for lol-html
mainBenchmarks can be executed usingcargo bench. You can filter specific benchmarks by providing a substring. After running benchmarks, the test report is generated and can be viewed attarget/criterion/report/index.html.Run fuzzing with various engines
mainThe project supports fuzzing through several engines. Note that
cargo-fuzzrequires a Rust nightly toolchain.cargo-fuzz (libFuzzer)
Use
./scripts/fuzz_with_libfuzzer.shfor the main crate, or./scripts/fuzz_c_api_with_libfuzzer.shspecifically for the C API.AFL
Use
./scripts/fuzz_with_afl.shto run fuzzing with AFL.honggfuzz
Use
./scripts/fuzz_with_hongg.shto run fuzzing with honggfuzz.Use the CSS selector VM's AST printer
mainThe CSS selector VM's AST printer allows you to inspect the selector VM's program AST for a list of CSS selectors. The selectors must be provided as a JSON array string.Run tests for lol-html
mainYou can run tests using standard Cargo commands or the provided project scripts. Usecargo testfor unit tests located in/src. For a comprehensive test suite that includes integration tests, C API tests, and linting, use the./scripts/test.shscript. You can filter tests by providing a substring in the command.Use the LOL HTML JavaScript API
mainThe
lol-htmlJavaScript API provides anHTMLRewriterclass for streaming HTML rewriter functionality. You can use it to intercept elements, modify attributes, and react to tag lifecycle events (likeonEndTag) while processing HTML chunks.To use it, instantiate
HTMLRewriterwith a character encoding and a callback function that handles output chunks. Use.on(selector, handlers)to define transformations for specific elements, and.write(buffer)to feed HTML data into the rewriter. Finally, call.end()to signal the end of the stream.'use strict'; const { HTMLRewriter } = require('lol-html'); // path/to/lol-html.js const chunks = []; const rewriter = new HTMLRewriter('utf8', (chunk) => { chunks.push(chunk); }); rewriter.on('a[href]', { element(el) { const href = el .getAttribute('href') .replace('http:', 'https:'); el.setAttribute('href', href); el.onEndTag((tag)=> { console.log(`Tag ended: ${tag.name}`); }); }, }); [ '<div><a href=', 'http://example.com>', '</a></div>', ].forEach((part) => { rewriter.write(Buffer.from(part)); }); rewriter.end(); const output = Buffer.concat(chunks).toString('utf8'); console.log(output);Build the JavaScript API from source
mainTo build the JavaScript API, you need
rustupandwasm-pack. This process compiles the Rust core to WebAssembly targeting Node.js.- Update your Rust toolchain using
rustup update. - Install
wasm-packusingcargo install wasm-pack. - Run the build command:
wasm-pack build --target nodejs --release.
rustup update # https://rustup.rs cargo install wasm-pack wasm-pack build --target nodejs --release- Update your Rust toolchain using
Set up the fuzzing environment
mainThe fuzzers in this repository require additional tools and specific environment configurations. They cannot be executed using onlycargo. To set up the necessary environment, refer to the scripts located in thescriptsdirectory at the root of the project.Use the HTML parser tracer
mainThe HTML parser tracer is a debugging tool that provides detailed trace information about the parsing process of a given HTML input. Use the./scripts/parser_trace.shscript to access it.Use the streaming HtmlRewriter API
mainThe
HtmlRewriteris the primary entry point for streaming HTML rewriting. It is designed for low-latency, minimal-buffering modifications to HTML documents, making it suitable for large files or memory-constrained environments. You provide handlers for different HTML components (like elements, text, or comments) and the rewriter processes the stream sequentially.Key components include:
HtmlRewriter: The main streaming rewriter.ElementHandler: To intercept and modify HTML elements.TextHandler: To intercept and modify text content.CommentHandler: To intercept and modify HTML comments.DoctypeHandler: To intercept and modify the DOCTYPE.EndHandler/EndTagHandler: To intercept the end of the document or specific tags.
For simple, non-streaming tasks, you can use the
rewrite_strfunction to perform one-off rewriting on a single HTML string.C API: Content Mutation and Streaming
mainThe C API provides functions for mutating HTML content and handling streaming output.
Content Mutation
Functions for content mutation (e.g., inserting text or elements) require:
- A valid, non-
NULLpointer to the mutation handler. content: A valid UTF-8 string.content_len: The length of the content.is_html: A boolean indicating if the content should be treated as HTML (written without escaping) or plain text.
Returns
0on success, or a non-zero error code on failure.Streaming
Streaming operations use the
CStreamingHandlerto provide callbacks for writing content.- The
CStreamingHandlermust be valid and non-NULL. - The
streaming_writeris copied immediately and is not guaranteed a stable address. - The
streaming_writercan be used from another thread (Send) but is not thread-safe for concurrent access (!Sync).
Returns
0on success, or-1if the handler is invalid or the writer isNULL.- A valid, non-
Rewrite mixed content using the mixed_content_rewriter example
mainThe
mixed_content_rewriterexample demonstrates how to read HTML from thestdinstream, rewrite mixed content (e.g., upgrading insecure URLs to secure ones) to prevent security issues, and stream the modified HTML tostdout.To run this example, you can pipe HTML content from a source like
curlinto the command viacargo run.curl -NL https://git.io/JeOSZ | cargo run --example=mixed_content_rewriter