DOCSEARCHbyAlgolia
Submit Docs

entities

repository·main·Indexed 18 days ago

https://github.com/fb55/entities

A high-performance library for encoding, decoding, and escaping HTML and XML entities. Version 8.0.0 provides specialized functions like escapeUTF8, encodeXML, and encodeHTML for speed and configurability. It features a memory-efficient radix tree mapped to a Uint16Array to reduce character reference data size and supports tree shaking for optimized bundle sizes. Includes an EntityDecoder for resumable/partial decoding and configurable DecodingModes (Legacy, Strict, Attribute) for HTML entity termination.

Tokens
5K
Snippets
18
Records
26
Agent score
64%

What's inside entities

  1. How semicolon handling works for HTML entities

    main

    The library handles the distinction between "strict" HTML entities (which require a semicolon) and "legacy" entities (where the semicolon is optional) using the node header:

    1. Strict Entities: The encoder does not emit an explicit ';' child node. Instead, it sets the semicolon-required flag (bit 13) when valueLength > 0. During decoding, the unsuffixed key is automatically replaced with the suffixed variant.
    2. Legacy Entities: These are represented as two separate nodes: one without the semicolon and one reached via an explicit ';' branch. These nodes do not set the semicolon-required flag.

    This mechanism ensures correct semantic behavior while optimizing the trie structure.

  2. Understand the Radix Tree structure in entities

    main

    The entities library uses a radix tree to encode named entities. This structure consists of nodes that contain both data (the entity value) and branches (the paths to child nodes). This approach is highly memory-efficient, reducing the size of character reference data from ~8.5Mb to ~250Kb compared to simple trie structures.

    In the trie, words share common prefixes. For example, the words test, tester, and testing would share the prefix t -> e -> s -> t, with branches splitting off at the final t to handle the suffixes er and ing or the terminal value for test.

  3. Understand branch data encoding modes

    main

    When a node has branches (children), the branch data immediately follows the node header (or the packed path in a compact run). There are three ways branches are represented:

    1. Single branch inlined: Used when there is exactly one child. The branch length bits (12..7) are set to 0, and the child character code is stored in bits 6..0. The child node header follows immediately.
    2. Jump table: Used when branch keys form a relatively dense range.
      • Bits 6..0 store the offset (minimum key).
      • Bits 12..7 store the span length (maxKey - minKey + 1).
      • A table of uint16 slots follows, where each slot stores destinationIndex + 1 (0 indicates no branch).
    3. Dictionary (sparse): Used for sparse or far-apart keys.
      • Packed key array: (branchCount + 1) >> 1 words, each containing two 8-bit sorted keys (low byte even index, high byte odd index).
      • Destination array: branchCount words, each containing a raw destination index.
      • The branch length bits store the branchCount, and the offset (bits 6..0) is 0 to distinguish it from the jump table mode.

    Note: Recursive or duplicated subtrees are deduplicated via node caching, meaning repeated branches point to the same encoded node index.

  4. How compact runs optimize the trie

    main

    To save space and reduce pointer chasing, the encoder uses "compact runs" for linear chains of nodes.

    A compact run occurs when a node has no value (valueLength == 0) and is part of a linear chain of at least three single-child nodes leading to a terminal or branching node.

    Compact Run Layout:

    • Bit 13: Set to the run flag.
    • Bits 12..7: Store the run length (6 bits, 1–63), representing the number of characters in the collapsed path.
    • Bits 6..0: Store the first character.
    • Following elements: The remaining (runLength - 1) characters are stored packed two per uint16 word (low byte / high byte) immediately after the header.
    • Final element: The child node that owned the value or branches is encoded in its normal form immediately after the packed characters.

  5. Optimize bundle size with tree shaking

    main
    The entities library supports tree shaking. To ensure the smallest possible bundle size, avoid using the generic encode and decode functions, as they wrap multiple internal functions that will all be included in your bundle. Instead, import and use the specific functions you need (e.g., escapeUTF8, decodeHTML) directly.

  6. How the trie is mapped to a Uint16Array

    main

    To avoid the high memory overhead of allocating JavaScript objects for every node, the trie is mapped to a single Uint16Array. This allows for extremely fast traversal and low memory consumption.

    Because the library deals with UTF-16 code points, a Uint16Array is used. While most code points fit in a single 16-bit word, surrogate pairs (used in some named character references) are handled by splitting them across two uint16 code points within the array.

  7. Choose the right encoding method

    main

    When deciding how to encode your documents, follow these guidelines:

    • If your target supports UTF-8: Use escapeUTF8. This is the most efficient method for saving bytes.
    • If targeting XML: Use encodeXML.
    • If targeting HTML: Use encodeHTML.

    You can further customize behavior by exploring the options available for the encode and decode methods.

  9. Configure the DecodingMode for HTML entities

    main

    The DecodingMode enum determines how strictly the decoder handles entity termination (semicolons) and how it behaves when parsing entities within HTML attributes.

    • DecodingMode.Legacy: The default mode. Allows entities in text nodes to end with any character (semicolon is optional).
    • DecodingMode.Strict: Only allows entities that are explicitly terminated with a semicolon.
    • DecodingMode.Attribute: Specifically for entities found within HTML attributes. It applies limitations on ending characters to match the HTML specification.
    export enum DecodingMode {
    /** Entities in text nodes that can end with any character. */
    Legacy = 0,
    /** Only allow entities terminated with a semicolon. */
    Strict = 1,
    /** Entities in attributes have limitations on ending characters. */
    Attribute = 2,
}

  10. Encode and decode HTML and XML entities

    main

    Use the entities module to transform text between raw characters and various entity formats.

    Encoding

    • escapeUTF8(text): Best choice if your target supports UTF-8. It minimizes byte size by keeping characters as UTF-8 instead of entities.
    • encodeXML(text): Encodes text specifically for XML documents.
    • encodeHTML(text): Encodes text specifically for HTML documents.

    Decoding

    • decodeXML(text): Decodes XML entities back to raw characters.
    • decodeHTML(text): Decodes HTML entities back to raw characters.
    import * as entities from "entities";

// Encoding
entities.escapeUTF8("& ü"); // "& ü"
entities.encodeXML("& ü"); // "& ü"
entities.encodeHTML("& ü"); // "& ü"

// Decoding
entities.decodeXML("asdf & ÿ ü '"); // "asdf & ÿ ü '"
entities.decodeHTML("asdf & ÿ ü '"); // "asdf & ÿ ü '"

  11. Node layout and header bit structure

    main

    Every node in the Uint16Array begins with a 16-bit header word. The bit layout is as follows:

    BitsPurpose
    15..14value length field: Encoded length of the value (using a "+1" scheme).
    13dual-use flag:
    - If valueLength > 0: semicolon-required flag (no explicit ';' branch is stored).
    - If valueLength == 0: compact run flag.
    12..7branch length / span: Meaning depends on the encoding mode (see "Branch data").
    6..0jump table offset OR first character (single branch/run) OR part of packed info.

    Value length encoding

    The 2-bit valueLength field uses a "+1" scheme:

    • 0: No value present.
    • 1: Single code unit value inlined in the lower 14 bits (bits 13..0). Note: The encoder ensures the 13th bit is not set for inlined characters to avoid collision.
    • 2: One code unit value stored in the next array element.
    • 3: Two code unit value stored in the next two array elements.
    15..14  value length field
13      dual-use flag
12..7   branch length / span
6..0    jump table offset OR first character OR part of packed info

  12. Encode strings for HTML attributes using `escapeAttribute`

    main

    Use escapeAttribute to encode characters required for HTML attributes according to the WHATWG HTML specification. It targets characters like quotes, ampersands, and non-breaking spaces.

    import { escapeAttribute } from 'entities';

const input = '"quoted" &  ';
const encoded = escapeAttribute(input);
// Result: ""quoted" &  "