simple-markdown

repository·master·Indexed 19 days ago

https://github.com/ariabuckles/simple-markdown

A Javascript markdown-like parser designed for simplicity and extensibility, used by Khan Academy to support custom extensions like math text and interactive widgets. It allows developers to define custom rules for matching, parsing, and rendering content into React elements or HTML strings. The library provides default block, inline, and implicit parsing APIs, as well as utilities for creating custom parsers and outputters via SimpleMarkdown.parserFor and SimpleMarkdown.outputFor.

Tokens
6.4K
Snippets
27
Records
34
Agent score
68%

What's inside simple-markdown

  1. How rules work in simple-markdown

    master

    In simple-markdown, elements are created from rules. A rule is an object that defines how to identify, parse, and render a specific markdown pattern.

    To create a custom extension, you must define a rule object with the following lifecycle methods:

    1. match(source, state): Determines if the current source matches the rule. It should return a capture object (like the result of RegExp.exec) if successful, or null if not. Note: Regexes used in match should always begin with ^ to avoid infinite loops.
    2. parse(capture, recurseParse, state): Transforms the capture into a syntax tree node. It uses recurseParse to handle nested content.
    3. react(node, recurseOutput, state) OR html(node, recurseOutput, state): Transforms the syntax node into the final output format. You typically implement one or the other depending on whether you are building a React or HTML outputter.
  2. Install and get started with simple-markdown

    master

    To use simple-markdown in a Node.js environment, install it via npm:

    npm install simple-markdown

    Then, require it to access the SimpleMarkdown object, which provides default parsers and outputters for generic markdown:

    var SimpleMarkdown = require("simple-markdown");
    
    // Get default parser and outputter
    var mdParse = SimpleMarkdown.defaultBlockParse;
    var mdOutput = SimpleMarkdown.defaultOutput;
    
    // Parse markdown into a syntax tree
    var syntaxTree = mdParse("Here is a paragraph and an *em tag*.");
    
    // Convert syntax tree to React elements
    var reactElements = mdOutput(syntaxTree);
    var SimpleMarkdown = require("simple-markdown");
    var mdParse = SimpleMarkdown.defaultBlockParse;
    var mdOutput = SimpleMarkdown.defaultOutput;
    var syntaxTree = mdParse("Here is a paragraph and an *em tag*.");
    var reactElements = mdOutput(syntaxTree);
  3. Implement a full parsing and output pipeline

    master

    To transform a raw string into React or HTML, follow these steps:

    1. Define Rules: Create a rules object by spreading SimpleMarkdown.defaultRules and overriding specific rules if needed.
    2. Initialize Parser: Create a parser using SimpleMarkdown.parserFor(rules).
    3. Initialize Outputter: Create an output function using SimpleMarkdown.outputFor(rules, 'react') or SimpleMarkdown.outputFor(rules, 'html').
    4. Process Input:
      • Append \n\n to the source string (many block rules require this trailing newline to trigger correctly).
      • Call the parser with {inline: false} to generate the syntax tree.
      • Pass the tree to the output function.
    var rules = {
        ...SimpleMarkdown.defaultRules,
        paragraph: {
            ...SimpleMarkdown.defaultRules.paragraph,
            react: (node, output, state) => {
                return <p key={state.key}>{output(node.content, state)}</p>;
            }
        }
    };
    
    var parser = SimpleMarkdown.parserFor(rules);
    var reactOutput = SimpleMarkdown.outputFor(rules, 'react');
    var htmlOutput = SimpleMarkdown.outputFor(rules, 'html');
    
    var blockParseAndOutput = function(source) {
        // Many rules require content to end in \n\n to be interpreted
        // as a block.
        var blockSource = source + "\n\n";
        var parseTree = parser(blockSource, {inline: false});
        var outputResult = htmlOutput(parseTree);
        // Or for react output, use:
        // var outputResult = reactOutput(parseTree);
        return outputResult;
    };
  4. Create a custom markdown extension

    master

    To add a custom extension (e.g., an underline rule __text__), follow these steps:

    1. Define a rule object with order, match, parse, and an output method (react or html).
    2. Extend the default rules using the new rule.
    3. Build a custom parser and outputter using SimpleMarkdown.parserFor and SimpleMarkdown.outputFor.
    // 1. Define the rule
    var underlineRule = {
      order: SimpleMarkdown.defaultRules.em.order - 0.5,
      match: function (source) {
        return /^__([SS]+?)__(?!_)/.exec(source);
      },
      parse: function (capture, parse, state) {
        return { content: parse(capture[1], state) };
      },
      html: function (node, output) {
        return "<u>" + output(node.content) + "</u>";
      }
    };
    
    // 2. Extend default rules
    var rules = _.extend({}, SimpleMarkdown.defaultRules, {
      underline: underlineRule,
    });
    
    // 3. Build custom parser/outputter
    var rawBuiltParser = SimpleMarkdown.parserFor(rules);
    var parse = function (source) {
      return rawBuiltParser(source + "\n\n", { inline: false });
    };
    var htmlOutput = SimpleMarkdown.outputFor(rules, "html");
    
    // Usage
    var tree = parse("__hello__");
    var html = htmlOutput(tree);
    var underlineRule = {
      order: SimpleMarkdown.defaultRules.em.order - 0.5,
      match: function (source) {
        return /^__([\s\S]+?)__(?!_)/.exec(source);
      },
      parse: function (capture, parse, state) {
        return { content: parse(capture[1], state) };
      },
      html: function (node, output) {
        return "<u>" + output(node.content) + "</u>";
      },
    };
    
    var rules = _.extend({}, SimpleMarkdown.defaultRules, {
      underline: underlineRule,
    });
    
    var rawBuiltParser = SimpleMarkdown.parserFor(rules);
    var parse = function (source) {
      var blockSource = source + "\n\n";
      return rawBuiltParser(blockSource, { inline: false });
    };
    var htmlOutput = SimpleMarkdown.outputFor(rules, "html");
    
    var syntaxTree = parse("__hello underlines__");
    var html = htmlOutput(syntaxTree);
  5. Implement custom output for HTML or React

    master

    Rules in simple-markdown can define how they are rendered into different formats by providing html or react properties.

    • HTML Output: Provide an html function that takes (node, nestedOutput, state) and returns a string.
    • React Output: Provide a react function that takes (node, nestedOutput, state) and returns a ReactElement.

    When outputting arrays of nodes (like in the Array rule), the library automatically handles grouping adjacent text nodes to prevent unnecessary fragmentation.

    // Example of a rule with both HTML and React output
    const myRule = {
      order: 5,
      match: inlineRegex(/\*\*/),
      parse: (capture, parse, state) => ({ type: 'bold', content: capture[0] }),
      html: (node, output, state) => `<strong>${output(node.content, state)}</strong>`,
      react: (node, output, state) => reactElement('strong', state.key, { children: output(node.content, state) })
    };
  6. Understand the Parsing State object

    master

    The state object is threaded through all match, parse, and output calls. It allows for stateful parsing and lookbehind.

    Key properties include:

    • state.inline: A boolean indicating if the current parsing context is inline.
    • state.prevCapture: Stores the full regex capture object from the previous match. This allows rules to implement limited lookbehind (e.g., checking if a list item follows another list item).
    • state.key: A numerical key used during output (especially for React) to provide stable keys for elements.
    • state._defs: A user-defined property (suggested) to store definitions like link references.
    • state._refs: Used internally to track reference nodes for later resolution.
  7. Create an outputter with SimpleMarkdown.outputFor(rules, key)

    master

    Use SimpleMarkdown.outputFor(rules, key) to generate a function that transforms a syntax tree into a specific output format.

    Parameters:

    • rules: The rules object used to define the output logic.
    • key: The property name in the rules object that defines the output type. Common values are 'react' or 'html'. You can provide a custom key if you are defining a custom output type.

    The returned function accepts a syntax tree node and a recursive output function, returning the rendered result for that node.

    var reactOutput = SimpleMarkdown.outputFor(rules, 'react');
    var htmlOutput = SimpleMarkdown.outputFor(rules, 'html');
  8. Use the default parsing and outputting APIs

    master

    The SimpleMarkdown object provides several pre-configured methods for common parsing needs:

    • SimpleMarkdown.defaultBlockParse(source): Returns a syntax tree assuming source is in a block scope (e.g., can contain paragraphs, lists, etc.).
    • SimpleMarkdown.defaultInlineParse(source): Returns a syntax tree assuming source is inline text. It does not emit <p> elements, making it useful for single-line fields.
    • SimpleMarkdown.defaultImplicitParse(source): Automatically determines scope. It parses as block if source ends with \n\n, otherwise it parses as inline.
    • SimpleMarkdown.defaultOutput(syntaxTree): Returns React-renderable output for a given syntaxTree.
  9. Create a parser with SimpleMarkdown.parserFor(rules)

    master

    Use SimpleMarkdown.parserFor(rules) to generate a parser function based on a provided rules object.

    Requirements for the rules object:

    • Each rule must contain a match and a parse function.
    • Each rule must have a numeric order field.

    The parser will process rules in order of increasing order values. If two rules have the same order, they are sorted lexicographically by their rule name.

    var parser = SimpleMarkdown.parserFor(rules);
  10. Access default rules via SimpleMarkdown.defaultRules

    master

    The SimpleMarkdown.defaultRules object contains the library's built-in rule definitions. Each rule is an object containing the following fields:

    • order: Numeric value used to determine the sequence of rule application.
    • match: Function used to identify the rule in the source text.
    • parse: Function used to transform matched text into a syntax tree node.
    • react: Function used to generate React components from a node.
    • html: Function used to generate HTML strings from a node.

    You can use this object to create custom rule sets by spreading the defaults and overriding specific rule properties.

    var rules = {
        ...SimpleMarkdown.defaultRules,
        paragraph: {
            ...SimpleMarkdown.defaultRules.paragraph,
            react: (node, output, state) => {
                return <p key={state.key}>{output(node.content, state)}</p>;
            }
        }
    };
  11. Reference: Rule method signatures

    master

    Detailed signatures for the methods within a rule object:

    match(source, state, [lookbehind])

    • source: The upcoming string starting at the current position.
    • state: A mutable state object. state.inline is true in inline scope and false/undefined in block scope.
    • lookbehind (Deprecated): The string previously captured. Use state.prevCapture instead.
    • Returns: A capture object (e.g., from RegExp.exec) if matched, otherwise null.

    parse(capture, recurseParse, state)

    • capture: The non-null result returned from match.
    • recurseParse: A function to recursively parse sub-content. Returns an array.
    • state: The mutable state object.
    • Returns: A node object. The type field is reserved and used for output.

    react(node, recurseOutput, state) / html(node, recurseOutput, state)

    • node: The object returned by parse.
    • recurseOutput: A function to recursively output sub-tree nodes.
    • state: The mutable state object.
  12. Use `parseInline` and `parseBlock` for scoped parsing

    master

    The parser can be manually scoped to either inline or block mode using these helper functions. This is useful when you are implementing a custom rule that needs to parse a specific subset of markdown (e.g., parsing the content inside a link or a blockquote).

    • parseInline(parse, content, state): Sets state.inline = true for the duration of the parse.
    • parseBlock(parse, content, state): Sets state.inline = false for the duration of the parse and ensures the content is padded with double newlines (\n\n).
    // Inside a custom rule's parse function:
    const contentAst = parseInline(nestedParse, capture[1], state);