streaming-markdown

repository·main·Indexed 18 days ago

https://github.com/thetarnav/streaming-markdown

An optimistic Markdown parser designed for streaming content, allowing developers to render markdown tokens to the DOM incrementally. It supports a wide range of features including basic formatting, lists, code blocks, tables, and LaTeX math. The library provides a default HTML renderer and a logger renderer for debugging, and allows for the implementation of custom Renderer interfaces.

Tokens
2.2K
Snippets
7
Records
11
Agent score
14%

What's inside streaming-markdown

  1. Supported Markdown features

    main

    The streaming-markdown parser supports a wide range of Markdown features, including:

    • Basic Formatting: Paragraphs, line breaks, headers, horizontal rules (---, ***, ___), blockquotes.
    • Emphasis: Italic (asterisks or underscores), Bold (double asterisks or underscores), Strikethrough.
    • Lists: Unordered, ordered (with start attribute), task lists, and nested lists.
    • Code: Inline code (`code`) and code blocks (triple backticks or indented), including language attributes.
    • Links & Media: Links ([text](url)), autolinks (e.g., https://...), and images (![alt](src)).
    • Tables: Standard markdown tables.
    • Math: LaTeX tags for blocks (\[...\], $$...$$) and inline (\(...\), $...$).
    • Other: Escaping characters, nested emphasis combinations, and <br> tags.
  2. Install streaming-markdown

    main

    You can install the streaming-markdown package via npm, copy the smd.js file directly into your project, or use a CDN link.

    If using the CDN, ensure your script tag uses type="module" because the package uses ES module exports.

    npm install streaming-markdown
    <script type="module">
        import * as smd from "https://cdn.jsdelivr.net/npm/streaming-markdown/smd.min.js"
        // ...
    </script>
  3. Basic usage of streaming-markdown

    main

    To use the library, follow these steps:

    1. Select a DOM element where the markdown should be rendered.
    2. Create a Renderer using one of the built-in functions (e.g., smd.default_renderer(element)).
    3. Initialize a Parser by passing the renderer to smd.parser(renderer).
    4. Stream content using smd.parser_write(parser, chunk).
    5. Finalize the stream using smd.parser_end(parser).

    The parser is optimistic: it styles elements like code blocks immediately upon seeing the opening delimiter, even before the closing delimiter arrives.

    import * as smd from "streaming-markdown"
    
    const element  = document.getElementById("markdown")
    const renderer = smd.default_renderer(element)
    const parser   = smd.parser(renderer)
    
    // Stream chunks of markdown
    smd.parser_write(parser, "# Streaming Markdown\n\n")
    smd.parser_write(parser, "## Next section...")
    
    // End the stream and flush remaining content
    smd.parser_end(parser)
  4. Finalize the stream with parser_end

    main
    The smd.parser_end(parser) function signals the end of the markdown stream. It resets the Parser state and flushes any remaining markdown tokens to ensure the final output is complete and correctly rendered.
    smd.parser_end(parser)
  5. Stream markdown content with parser_write

    main

    The smd.parser_write(parser, chunk) function allows you to feed chunks of a markdown string into the parser. You can call this function multiple times to simulate a real-time stream (like a ChatGPT response). The parser only adds new elements to the DOM and does not modify existing ones, allowing users to select and copy text that has already been rendered.

    smd.parser_write(parser, "# Streaming Markdown\n\n")
  6. Implement a custom Renderer interface

    main

    A Renderer is an object that defines how parsed markdown tokens are converted into DOM elements or other outputs. To create a custom renderer, implement the following interface:

    Field nameTypeDescription
    dataTUser data object. Available as the first parameter in callbacks.
    add_tokenRenderer_Add_Token<T>Called when a token starts.
    end_tokenRenderer_End_Token<T>Called when a token ends.
    add_textRenderer_Add_Text<T>Appends text to the current token. Can be called zero or more times.
    set_attrRenderer_Set_Attr<T>Sets additional attributes on the current token (e.g., a link's href).
  7. Use the logger renderer for debugging

    main

    If you need to inspect the sequence of tokens and text being processed by the parser, use logger_renderer(). This implementation does not produce output but logs all parser actions to the console:

    • add_token: Logs add_token: <TOKEN_NAME>
    • end_token: Logs end_token
    • add_text: Logs add_text: "%s"
    • set_attr: Logs set_attr: <attr>=<value>
    import { logger_renderer } from './smd.js';
    
    const renderer = logger_renderer();
    // Use this renderer to see the parser's lifecycle in the console.
  8. Use the default HTML renderer

    main

    The default_renderer(root) function provides a ready-to-use implementation that converts Markdown tokens directly into DOM elements inside a provided HTMLElement root. It manages a stack of nodes using an internal index and nodes array within its data object.

    When using the default renderer, add_token creates elements like <h1>, <p>, <em>, or <a>, and add_text appends text nodes to the current element.

    import { default_renderer } from './smd.js';
    
    const root = document.getElementById('output');
    const renderer = default_renderer(root);
    
    // The renderer is now ready to be used by the parser
    // e.g., parser.write(text, renderer);
  9. Reference: Default Renderer Token Mappings

    main

    The default_renderer maps specific Markdown tokens to the following HTML elements:

    TokenHTML Element
    BLOCKQUOTE<blockquote>
    PARAGRAPH<p>
    LINE_BREAK<br>
    RULE<hr>
    HEADING_1 to HEADING_6<h1> to <h6>
    ITALIC_AST / ITALIC_UND<em>
    STRONG_AST / STRONG_UND<strong>
    STRIKE<s>
    CODE_INLINE<code>
    RAW_URL / LINK<a>
    IMAGE<img>
    LIST_UNORDERED<ul>
    LIST_ORDERED<ol>
    LIST_ITEM<li>
    CHECKBOX<input type="checkbox" disabled>
    CODE_BLOCK / CODE_FENCE<pre><code>
    TABLE<table>
    TABLE_ROW<tr> (inside <thead> or <tbody>)
    TABLE_CELL<th> (in head) or <td> (in body)
    EQUATION_BLOCK<equation-block>
    EQUATION_INLINE<equation-inline>
  10. Reference the Token enum for Markdown types

    main

    The Token enum provides integer constants representing the different Markdown elements identified by the parser. Use these constants when implementing a custom Renderer or when inspecting token types.

    export const Token = {
        Document:       1,
        Blockquote:     20,
        Paragraph:      2,
        Heading_1:      3,
        Heading_2:      4,
        Heading_3:      5,
        Heading_4:      6,
        Heading_5:      7,
        Heading_6:      8,
        Code_Block:     9,
        Code_Fence:     10,
        Code_Inline:    11,
        Italic_Ast:     12,
        Italic_Und:     13,
        Strong_Ast:     14,
        Strong_Und:     15,
        Strike:         16,
        Link:           17,
        Raw_URL:        18,
        Image:          19,
        Line_Break:     21,
        Rule:           22,
        List_Unordered: 23,
        List_Ordered:   24,
        List_Item:      25,
        Checkbox:       26,
        Table:          27,
        Table_Row:      28,
        Table_Cell:     29,
        Equation_Block: 30,
        Equation_Inline: 31,
    }