DOCSEARCHbyAlgolia
Submit Docs

MD4X Documentation

repository·main·Indexed 18 days ago

https://github.com/unjs/md4x

A high-performance, small-footprint markdown parser and renderer written in C. It supports Node.js via NAPI and universal environments via WASM, featuring a SAX-like streaming parser, Comark AST support, and built-in healing for incomplete streaming LLM output. MD4X provides a CLI and libraries to render markdown into HTML, ANSI, plain text, JSON AST, and metadata.

Tokens
25.3K
Snippets
105
Records
114
Agent score
62%

What's inside md4x

  1. Supported Markdown Syntax in md4x

    main

    md4x is a fast markdown parser and renderer that supports a wide range of standard Markdown features, including:

    • Inline Formatting: Bold (**), Italic (_), Underline, Strikethrough (~~), and Inline Code ( ).
    • Links & Autolinks: Explicit links [text](url), and automatic detection of URLs, emails, and www. addresses.
    • Images: Standard markdown image syntax ![alt](url).
    • Lists: Unordered lists (-) and Ordered lists (1.).
    • Task Lists: Checkable items using - [x] for completed and - [ ] for incomplete.
    • Blockquotes: Nested blockquotes using >.
    • Horizontal Rules: Separators using ---.
    • Hard Breaks: Triggered by two trailing spaces or a backslash \ at the end of a line.

  3. Understand the Comark AST structure

    main

    The Comark AST (Abstract Syntax Tree) is a lightweight, array-based format designed for efficiency. Instead of traditional object-based nodes, it uses nested arrays (tuples) to represent the document structure. This makes it memory-efficient and predictable for processing.

    Core Types

    • ComarkTree: The root object containing nodes (the parsed content), frontmatter (YAML data), and meta (plugin-specific metadata like TOC or summaries).
    • ComarkNode: A union type that can be either a ComarkText (a plain string) or a ComarkElement (a tuple array).
    • ComarkText: A simple string representing text content.
    • ComarkElement: A tuple array in the format [tag, props, ...children].
    interface ComarkTree {
  nodes: ComarkNode[];
  frontmatter: Record<string, any>;
  meta: {
    toc?: any;
    summary?: ComarkNode[];
    [key: string]: any;
  };
}

type ComarkNode = ComarkElement | ComarkText;
type ComarkText = string;
type ComarkElement = [string, ComarkElementAttributes, ...ComarkNode[]];
type ComarkElementAttributes = {
  [key: string]: unknown;
};

  4. Understand the Comark AST format

    main

    The renderToAST method produces a Comark AST, a lightweight, array-based JSON format. The root object contains nodes, frontmatter, and meta.

    Node Structure:

    • Text Node: A plain string.
    • Element Node: A tuple array: [tag: string | null, props: ComarkElementAttributes, ...children: ComarkNode[]].
    • HTML Comments: Represented as [null, {}, " comment text "].

    Property Mappings:

    • prop="value" $\rightarrow$ "prop": "value" (String)
    • bool $\rightarrow$ ":bool": "true" (Boolean with : prefix)
    • :count="5" $\rightarrow$ ":count": "5" (Number/bind with : prefix)
    • :data='{"k":"v"}' $\rightarrow$ ":data": "{\"k\":\"v\"}" (JSON passthrough)
    • #my-id $\rightarrow$ "id": "my-id" (ID shorthand)
    • .class-one .class-two $\rightarrow$ "class": "class-one class-two" (Class shorthand)

    Example Code Block Node: ["pre", {"language": "js", "filename": "app.js", "highlights": [1,2]}, ["code", {"class": "language-js"}, "..."]]

  5. Use MD4X parser dialects

    main

    Instead of configuring individual flags, you can use predefined compound dialects to quickly set up common parsing environments:

    • MD_DIALECT_COMMONMARK (Value: 0): Strict CommonMark compliance.
    • MD_DIALECT_GITHUB: A permissive set including autolinks, tables, strikethrough, task lists, and alerts.
    • MD_DIALECT_ALL: Enables all available additive extensions (autolinks, tables, strikethrough, tasklists, latex math, wikilinks, underline, frontmatter, components, attributes, and alerts).
    // Example of using a dialect
// MD_DIALECT_GITHUB = permissive autolinks + tables + strikethrough + task lists + alerts
// MD_DIALECT_ALL = all additive extensions

  6. Heal incomplete markdown for streaming LLM output

    main

    MD4X provides a heal() function and a { heal: true } option to fix incomplete markdown syntax (like unclosed bold, italics, or code blocks). This is ideal for real-time rendering of streaming LLM outputs.

    Using heal() directly

    import { heal } from "md4x";

heal("**bold"); // returns "**bold**"
heal("```js\ncode"); // returns "```js\ncode\n```"

    Healing during rendering

    Pass { heal: true } as an option to any render function to heal the input in a single pass before rendering.

    import { renderToHtml } from "md4x";

// Heal + render in one call
renderToHtml("# Hello **world", { heal: true });
// "<h1>Hello <strong>world</strong></h1>\n"
    import { heal } from "md4x";

heal("**bold"); // "**bold**"
heal("*ita"); // "*ita*"
heal("~~strike"); // "~~strike~~"
heal("`code"); // "`code`"
heal("```js\ncode"); // "```js\ncode\n```"
heal("[text](http:"); // ""  (strips broken links)

// Using the option in render functions
import { renderToHtml, parseAST, renderToAnsi, renderToText } from "md4x";

renderToHtml("# Hello **world", { heal: true });
parseAST("# Hello **world", { heal: true });
renderToAnsi("# Hello **world", { heal: true });
renderToText("# Hello **world", { heal: true });

  7. Format of Comark Element Nodes

    main

    Every ComarkElement follows a strict tuple format at specific indices:

    • Index 0 (tag): The element name (e.g., "p", "h1") or a component tag.
    • Index 1 (props): An object containing attributes/properties (e.g., { "href": "..." }).
    • Index 2+ (children): The subsequent elements in the array are the child nodes, which can be either strings (ComarkText) or nested arrays (ComarkElement).

    Note on Comments: HTML comments are represented by using null as the tag at index 0: [null, {}, " comment text "].

    // Example: <p>This is a paragraph</p>
["p", {}, "This is a paragraph"]

// Example: <a href="https://example.com">Link</a>
["a", { "href": "https://example.com" }, "Link"]

  8. Understand the MD_ATTRIBUTE structure

    main

    The MD_ATTRIBUTE struct is used for non-text-flow content like URLs, titles, or component props. It handles strings that might contain a mix of normal text and HTML entities.

    Structure:

    • text: Pointer to the raw character data.
    • size: Total size of the attribute.
    • substr_types: An array of MD_TEXTTYPE describing the segments within the string.
    • substr_offsets: An array of offsets into the text buffer marking the start of each segment.

    Invariants:

    • substr_offsets[0] is always 0.
    • substr_offsets[LAST+1] is always equal to size.
    • Only MD_TEXT_NORMAL, MD_TEXT_ENTITY, and MD_TEXT_NULLCHAR are permitted as substring types.
    typedef struct MD_ATTRIBUTE {
    const MD_CHAR* text;
    MD_SIZE size;
    const MD_TEXTTYPE* substr_types;    /* Array of substring types */
    const MD_OFFSET* substr_offsets;    /* Array of substring offsets */
}

  9. Traverse, find, and modify Comark AST nodes

    main

    Since the AST is array-based, you can manipulate it using standard array methods and recursion.

    Traversing Nodes

    To visit every node, check if a node is an array. If it is, the children start at index 2.

    Finding Elements

    To find specific elements (e.g., all h1 tags), recursively search the nodes array and check if node[0] === tag.

    Modifying Nodes

    When modifying, it is best practice to treat the AST as immutable. Create new arrays/objects instead of mutating existing ones to avoid side effects.

    // Traversing example
function traverse(node: ComarkNode, callback: (node: ComarkNode) => void) {
  callback(node);
  if (Array.isArray(node)) {
    const children = node.slice(2);
    for (const child of children) {
      traverse(child, callback);
    }
  }
}

// Modifying example (adding a class to all links)
function addClassToLinks(node: ComarkNode): ComarkNode {
  if (Array.isArray(node)) {
    const [tag, props, ...children] = node;
    if (tag === "a") {
      return [tag, { ...props, class: "external-link" }, ...children];
    }
    return [tag, props, ...children.map(addClassToLinks)];
  }
  return node;
}

  10. Use LaTeX Math in MD4X

    main

    MD4X supports LaTeX math syntax for rendering mathematical equations.

    • Inline math: Wrap expressions in single dollar signs (e.g., $E = mc^2$).
    • Display math: Wrap expressions in double dollar signs on separate lines for centered, block-level equations (e.g., $$\int_0^\infty e^{-x^2} dx = \frac{\sqrt{\pi}}{2}$$).
    Inline math: $E = mc^2$

Display math:

$$
\int_0^\infty e^{-x^2} dx = \frac{\sqrt{\pi}}{2}
$$

  11. Apply Inline Attributes and Span Syntax

    main

    You can apply attributes directly to text elements using curly braces {}. This includes classes, IDs, and other properties.

    • Classes: **bold**{.highlight}
    • IDs: _italic_{#myid}
    • Language tags: `code`{.lang-ts}
    • Links: [Link](url){target="_blank" .external}
    • Spans: [Styled text]{.primary}
    **bold**{.highlight}
_italic_{#myid}
`code`{.lang-ts}
[Link](https://github.com){target="_blank" .external}
[Styled text]{.primary}
[Important]{#notice .badge style="color: red" data-priority="high"}

  12. Use Block Components in Comark

    main

    Block components are defined using double colons (::) and wrap a block of content. They can accept properties via curly braces {} and support slots for structured content.

    • Standard block: ::component-name{prop="value"}
    • With slots: Use #header, #content, or #footer within the block to assign content to specific slots.
    • Closing syntax: Use :: to close a block.
    ::card{title="Features"}

- Fast parsing
- Low memory
::

::card
#header
Card Header

#content
Main content.

#footer
Card Footer
::