lol-html

repository·main·Indexed 24 days ago

https://github.com/cloudflare/lol-html

A low-latency, streaming HTML rewriter and parser with a CSS selector-based API. Designed for minimal memory overhead and buffering, it allows developers to modify HTML content on the fly. It is available as a Rust library, with a JavaScript API via WebAssembly, and official C bindings, as well as unofficial bindings for Lua, Go, and Ruby.

Tokens
6.2K
Snippets
14
Records
34
Agent score
85%

What's inside lol-html

  1. What is LOL HTML

    main
    LOL HTML (Low Output Latency streaming HTML) is a streaming HTML rewriter and parser that uses a CSS-selector based API. It is designed to modify HTML on the fly with minimal buffering, making it suitable for very large documents and environments with limited memory resources. It serves as the backend for HTML rewriting in Cloudflare Workers but is available as a standalone library.
  2. Run benchmarks for lol-html

    main
    Benchmarks can be executed using cargo bench. You can filter specific benchmarks by providing a substring. After running benchmarks, the test report is generated and can be viewed at target/criterion/report/index.html.
  3. Run fuzzing with various engines

    main

    The project supports fuzzing through several engines. Note that cargo-fuzz requires a Rust nightly toolchain.

    cargo-fuzz (libFuzzer)

    Use ./scripts/fuzz_with_libfuzzer.sh for the main crate, or ./scripts/fuzz_c_api_with_libfuzzer.sh specifically for the C API.

    AFL

    Use ./scripts/fuzz_with_afl.sh to run fuzzing with AFL.

    honggfuzz

    Use ./scripts/fuzz_with_hongg.sh to run fuzzing with honggfuzz.

  4. Run tests for lol-html

    main
    You can run tests using standard Cargo commands or the provided project scripts. Use cargo test for unit tests located in /src. For a comprehensive test suite that includes integration tests, C API tests, and linting, use the ./scripts/test.sh script. You can filter tests by providing a substring in the command.
  5. Use the LOL HTML JavaScript API

    main

    The lol-html JavaScript API provides an HTMLRewriter class for streaming HTML rewriter functionality. You can use it to intercept elements, modify attributes, and react to tag lifecycle events (like onEndTag) while processing HTML chunks.

    To use it, instantiate HTMLRewriter with a character encoding and a callback function that handles output chunks. Use .on(selector, handlers) to define transformations for specific elements, and .write(buffer) to feed HTML data into the rewriter. Finally, call .end() to signal the end of the stream.

    'use strict';
    
    const { HTMLRewriter } = require('lol-html'); // path/to/lol-html.js
    
    const chunks = [];
    const rewriter = new HTMLRewriter('utf8', (chunk) => {
      chunks.push(chunk);
    });
    
    rewriter.on('a[href]', {
      element(el) {
        const href = el
          .getAttribute('href')
          .replace('http:', 'https:');
        el.setAttribute('href', href);
    
        el.onEndTag((tag)=> {
          console.log(`Tag ended: ${tag.name}`);
        });
      },
    });
    
    [
      '<div><a href=',
      'http://example.com>',
      '</a></div>',
    ].forEach((part) => {
      rewriter.write(Buffer.from(part));
    });
    
    rewriter.end();
    
    const output = Buffer.concat(chunks).toString('utf8');
    console.log(output);
  6. Build the JavaScript API from source

    main

    To build the JavaScript API, you need rustup and wasm-pack. This process compiles the Rust core to WebAssembly targeting Node.js.

    1. Update your Rust toolchain using rustup update.
    2. Install wasm-pack using cargo install wasm-pack.
    3. Run the build command: wasm-pack build --target nodejs --release.
    rustup update # https://rustup.rs
    cargo install wasm-pack
    
    wasm-pack build --target nodejs --release
  7. Set up the fuzzing environment

    main
    The fuzzers in this repository require additional tools and specific environment configurations. They cannot be executed using only cargo. To set up the necessary environment, refer to the scripts located in the scripts directory at the root of the project.
  8. Use the streaming HtmlRewriter API

    main

    The HtmlRewriter is the primary entry point for streaming HTML rewriting. It is designed for low-latency, minimal-buffering modifications to HTML documents, making it suitable for large files or memory-constrained environments. You provide handlers for different HTML components (like elements, text, or comments) and the rewriter processes the stream sequentially.

    Key components include:

    • HtmlRewriter: The main streaming rewriter.
    • ElementHandler: To intercept and modify HTML elements.
    • TextHandler: To intercept and modify text content.
    • CommentHandler: To intercept and modify HTML comments.
    • DoctypeHandler: To intercept and modify the DOCTYPE.
    • EndHandler / EndTagHandler: To intercept the end of the document or specific tags.

    For simple, non-streaming tasks, you can use the rewrite_str function to perform one-off rewriting on a single HTML string.

  9. C API: Content Mutation and Streaming

    main

    The C API provides functions for mutating HTML content and handling streaming output.

    Content Mutation

    Functions for content mutation (e.g., inserting text or elements) require:

    • A valid, non-NULL pointer to the mutation handler.
    • content: A valid UTF-8 string.
    • content_len: The length of the content.
    • is_html: A boolean indicating if the content should be treated as HTML (written without escaping) or plain text.

    Returns 0 on success, or a non-zero error code on failure.

    Streaming

    Streaming operations use the CStreamingHandler to provide callbacks for writing content.

    • The CStreamingHandler must be valid and non-NULL.
    • The streaming_writer is copied immediately and is not guaranteed a stable address.
    • The streaming_writer can be used from another thread (Send) but is not thread-safe for concurrent access (!Sync).

    Returns 0 on success, or -1 if the handler is invalid or the writer is NULL.

  10. Rewrite mixed content using the mixed_content_rewriter example

    main

    The mixed_content_rewriter example demonstrates how to read HTML from the stdin stream, rewrite mixed content (e.g., upgrading insecure URLs to secure ones) to prevent security issues, and stream the modified HTML to stdout.

    To run this example, you can pipe HTML content from a source like curl into the command via cargo run.

    curl -NL https://git.io/JeOSZ | cargo run --example=mixed_content_rewriter