Lezer Markdown Parser

repository·main·Indexed 41 days ago

https://github.com/lezer-parser/markdown

An incremental Markdown parser (version 1.6.3) that produces Lezer-style syntax trees. It supports CommonMark and various extensions, including GitHub Flavored Markdown (GFM), and is designed for high-performance integration with the Lezer parser system and CodeMirror editors.

Tokens
7.4K
Snippets
27
Records
37
Agent score
100%

What's inside Lezer Markdown

  1. Overview of @lezer/markdown

    main

    @lezer/markdown is an incremental Markdown parser that supports CommonMark and various extensions. It is designed to integrate with the Lezer parser system by producing Lezer-style compact syntax trees and consuming fragments of such trees for incremental parsing.

    Key Characteristics:

    • Incremental Parsing: It is designed for performance by only re-parsing changed parts of a document.
    • Syntax Tree Focus: It produces a data structure representing the syntactic form of the document. It does not handle HTML output.
    • Parsing Limitations: To maintain single-pass incremental performance, it does not perform certain CommonMark validations, such as validating link references (e.g., [a][b] will be parsed as a link even if the reference [b] is missing).
    • CodeMirror Integration: For editor support, use the @codemirror/lang-markdown package, which integrates this parser into the CodeMirror editor.
  2. Implement a BlockParser

    main

    Block parsers handle the structural level of Markdown. There are three types of BlockParser implementations:

    1. Composite block parsers: Handle structures like lists or blockquotes. They implement a parse method that calls cx.startComposite(...) and returns null when the syntax is recognized.
    2. Eager leaf block parsers: Handle unambiguous blocks like code or HTML. Their parse method recognizes the construct, calls cx.nextLine() to move past the block, calls cx.addElement(...), and returns true.
    3. Leaf block parsers: Observe paragraph-like constructs (e.g., setext headings or link references). They implement a leaf method that returns a LeafBlockParser object if the parser wants to observe the current block.
    interface BlockParser {
      name: string;
      parse?: (cx: BlockContext, line: Line) => boolean | null;
      leaf?: (cx: BlockContext, line: Line) => LeafBlockParser;
    }
  3. Use DelimiterType for automatic matching

    main

    Delimiters are used during inline parsing to store positions of potential delimiters. They are identified by the DelimiterType interface.

    There are two ways to handle delimiters:

    1. Automatic Matching: Provide a resolve property. When a piece of inline content finishes, the system automatically matches delimiters of the same type (based on their open and close properties). The content between them is wrapped in a node named by the resolve value.
    2. Manual Matching: If resolve is not provided, you must match the delimiter eagerly using findOpeningDelimiter and takeContent methods on the InlineContext.
  4. Configure Block Parser installation order

    main

    When defining a block parser, you can control its position in the parsing hierarchy using the before or after options. This allows you to insert custom parsers relative to existing standard Markdown parsers.

    Available standard parser names for positioning include:

    • LinkReference
    • IndentedCode
    • FencedCode
    • Blockquote
    • HorizontalRule
    • BulletList
    • OrderedList
    • ATXHeading
    • HTMLBlock
    • SetextHeading
  5. Configure the Markdown parser with MarkdownConfig

    main

    The MarkdownConfig interface is used to customize the behavior of the MarkdownParser via the configure method. You can provide a single MarkdownConfig object or a nested array of them (a MarkdownExtension).

    Available configuration keys:

    • props?: Readonly NodePropSource[] to add node properties to the parser's node set.
    • defineNodes?: Readonly (string | NodeSpec)[] to define new syntax node types.
    • parseBlock?: Readonly BlockParser[] to define additional block parsing logic.
    • parseInline?: Readonly InlineParser[] to define new inline parsing logic.
    • remove?: Readonly string[] to remove named parsers from the configuration.
    • wrap?: A ParseWrapper to add a parse wrapper (e.g., for mixed-language parsing).

    To define new node types, use the NodeSpec interface, which requires a name (string) and can optionally include a block (boolean) property.

  6. Use GitHub Flavored Markdown (GFM) extensions

    main

    The GFM constant is an extension bundle containing several GFM-style features. You can pass this directly to MarkdownParser.configure().

    Included extensions:

    • Table: Supports GFM-style tables.
    • TaskList: Supports task list items with [ ] or [x] prefixes.
    • Strikethrough: Supports ~~ delimiters.
    • Autolink: Supports autolinking for www., http://, https://, mailto:, and xmpp: URLs and email addresses.
    const gfmParser = parser.configure(GFM);
  7. Create a Markdown extension for nested parsing with parseCode

    main

    Use parseCode to create a MarkdownExtension that enables nested parsing within code blocks or embedded HTML.

    Configuration options for parseCode:

    • codeParser?: A function (info: string) => Parser | null. The info argument is the string following the opening ``` marker (or an empty string for indented code blocks). If a parser is available, this function should return a parser that can construct a PartialParse.
    • htmlParser?: A Parser used to parse HTML tags (both block and inline).
    const codeExtension = parseCode({
      codeParser: (info) => {
        if (info === 'javascript') return javascriptParser;
        return null;
      },
      htmlParser: htmlParser
    });
  8. Use the Line class for block-level parsing

    main

    The Line class is a data structure used during block-level, per-line parsing. It provides access to the line's text, indentation, and positioning information, along with utility methods for navigating the line.

    Key Properties

    • text: The full string content of the line.
    • baseIndent: The base indentation provided by composite contexts.
    • basePos: The string position corresponding to the base indent.
    • pos: The position of the next non-whitespace character after composite block markers (like list markers or blockquotes).
    • indent: The column of the next non-whitespace character.
    • next: The character code of the character immediately following pos.

    Key Methods

    • skipSpace(from: number) => number: Returns the position of the next non-space character after from (or the end of the line).
    • moveBase(to: number): Moves the line's base position forward to the specified position. (Used by composite parsers).
    • moveBaseColumn(indent: number): Moves the line's base position forward to the specified column.
    • addMarker(elt: Element): Stores a composite-block-level marker. Should be called from markup skipping functions.
    • countIndent(to: number, from?: number, indent?: number) => number: Finds the column position at to.
  9. Use the MarkdownParser class

    main

    The MarkdownParser is the default CommonMark parser. It extends the base Lezer Parser and provides methods for reconfiguration and inline parsing.

    Key methods:

    • configure(spec: MarkdownExtension): Returns a new MarkdownParser instance configured with the provided extension(s).
    • parseInline(text: string, offset: number): Parses inline text at a specific offset, returning an array of Element objects representing the inline content.
    // Note: The exact instantiation depends on the exported 'parser' instance
    // which is the default CommonMark parser.
    const newParser = parser.configure(someExtension);
  10. Implement an InlineParser

    main

    An InlineParser is used to handle inline syntax during the parsing process. It must implement a parse method that receives the current InlineContext, the next character, and the current position.

    Behavior requirements:

    • Return -1 if the parser does not handle the current character.
    • If it handles the character, it should add an Element or a Delimiter to the context and return the end position of the parsed content.

    Ordering parsers: When defining an inline parser, you can control its precedence using:

    • before: Installs the parser directly before a parser with the specified name.
    • after: Installs the parser directly after a parser with the specified name.
    • If neither is provided, the parser is added to the end of the list.

    Default parser names include: Escape, Entity, InlineCode, HTMLTag, Emphasis, HardBreak, Link, and Image.

    parse(cx: InlineContext, next: number, pos: number) => number
  11. Implement a LeafBlockParser to override paragraph blocks

    main

    If a line does not match any regular block parsers, its content is typically accumulated into a paragraph-style block. To override this behavior, you can provide a leaf function in your parser configuration that returns a LeafBlockParser object.

    An object conforming to the LeafBlockParser interface must implement:

    • nextLine(cx: BlockContext, line: Line, leaf: LeafBlock) => boolean: Updates the parser state for the next line. Returning true finishes the block. This is called for all lines after the first.
    • finish(cx: BlockContext, leaf: LeafBlock) => boolean: Called when the block is finished by external circumstances (like a blank line). If the parser can handle the block up to its current position, it should finish the block and return true.
  12. Add highlighting tags to a NodeSpec

    main

    The style property in a NodeSpec allows you to assign highlighting information to a node. It accepts:

    • A single Tag.
    • An array of Tags.
    • An object following the styleTags pattern (similar to CodeMirror's styleTags argument) for complex rules.
    style?: Tag | readonly Tag[] | { [key: string]: Tag | readonly Tag[] }