Parsimmon

repository·master·Indexed 23 days ago

https://github.com/jneen/parsimmon

A monadic LL(infinity) parser combinator library for JavaScript (version 1.18.1) inspired by Parsec and compatible with Fantasyland protocols. It allows developers to build complex parsers by composing smaller, simpler ones using tools like Parsimmon.createLanguage, Parsimmon.seq, and Parsimmon.alt. The library supports both string and binary data parsing via the Parsimmon.Binary namespace for Node.js Buffers.

Tokens
8.9K
Snippets
21
Records
43
Agent score
77%

What's inside parsimmon

  1. What is Parsimmon?

    master
    Parsimmon is a parser combinator library designed for writing large parsers by composing many small parsers. Its API is inspired by parsec and Promises/A+. It is compatible with Fantasyland specifications, implementing Semigroup, Apply, Applicative, Functor, Chain, and Monad.
  2. Avoid negative constructions; use .notFollowedBy() instead

    master

    Parsimmon does not provide a Parsimmon.not combinator because inverting a parser's success/failure makes it difficult to report meaningful error messages and creates ambiguity regarding how much input should be consumed.

    If you need to ensure a pattern does not follow a certain sequence without consuming input, use .notFollowedBy() or Parsimmon.notFollowedBy().

  3. Avoid side effects in parsers

    master

    Parsimmon parsers and .map() statements must be pure. Do not perform side effects such as:

    • Pushing to an external array.
    • Modifying objects.
    • console.log.
    • Reading from external data sources.
    • Generating random numbers.

    Why: Parsimmon uses backtracking (e.g., via Parsimmon.alt). If a parser performs a side effect and then fails, Parsimmon will backtrack to try an alternative, but the side effect cannot be undone. This leads to incorrect state (e.g., duplicate entries in an array).

  4. Use Parsimmon.Binary constructors for Buffer parsing

    master

    The Parsimmon.Binary namespace provides constructors for parsing binary content using Node.js Buffers. These can be combined with standard combinators like Parsimmon.seq or Parsimmon.seqObj and support methods like .map() and .node().

    Common binary parsers include:

    • Parsimmon.Binary.byte(int): Matches a specific byte.
    • Parsimmon.Binary.buffer(length): Consumes a specific number of bytes and returns them as a cloned Buffer.
    • Parsimmon.Binary.encodedString(encoding, length): Parses length bytes and decodes them using the specified encoding (e.g., 'utf8').
    • Parsimmon.Binary.uint8, int8, uint16BE, int16LE, uint32BE, int32LE, etc.: Standard integer and float parsers for various bit-widths and endianness.
    // Example: Parsing a specific byte
    var parser = Parsimmon.Binary.byte(0x3f);
    parser.parse(Buffer.from([0x3f]));
    // => { status: true, value: 63 }
    
    // Example: Parsing an encoded string
    var parser = Parsimmon.Binary.encodedString("utf8", 17);
    parser.parse(Buffer.from([
      0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x74, 0x68, 0x65, 0x72, 0x65, 0x21, 0x20, 0xf0, 0x9f, 0x98, 0x84
    ]));
    // => { status: true, value: 'hello there! 😄' }
  5. Implement recursive parsers with Parsimmon.lazy(fn)

    master

    Accepts a function that returns a parser, which is evaluated the first time the parser is used. This is essential for implementing recursive parsers or referencing parsers that haven't been defined yet.

    Note: If you are using Parsimmon.createLanguage, Parsimmon.lazy is typically not needed.

    var Value = Parsimmon.lazy(function() {
      return Parsimmon.alt(
        Parsimmon.string("X"),
        Parsimmon.string("(")
          .then(Value)
          .skip(Parsimmon.string(")"))
      );
    });
    
    Value.parse("X"); // => {status: true, value: 'X'}
    Value.parse("(X)"); // => {status: true, value: 'X'}
    Value.parse("((X))"); // => {status: true, value: 'X'}
  6. Parse indentation-sensitive languages

    master

    To parse languages like Python or Markdown that rely on indentation (nesting structure), use Parsimmon.createLanguage inside a constructor function. This allows you to pass context (like indentSize) and generate new language instances with updated indentation levels as you descend into nested blocks.

    const createMyLanguage = ({ indentSize }) =>
      Parsimmon.createLanguage({
        Indent: () => Parsimmon.string(" ").times(indentSize),
        ForLoop: l =>
          l.SomeBlockStart.chain(block => {
            const lang = createMyLanguage({
              indentSize: block.newIndentSize
            });
            return lang.Item.atLeast(1);
          })
      });
    
    const MyLanguage = createMyLanguage({ indentSize: 0 });
    const ast = MyLanguage.File.tryParse(/* ... */);
  7. Understand Parsimmon terminology

    master

    To use Parsimmon effectively, understand these three core concepts:

    • Yields: When a function is said to yield a value (e.g., an array of strings), it means the resulting parser, when executed via .parse(), will return an object containing that value.
    • Input: The string provided to the .parse() method is referred to as the input.
    • Consumes: A parser consumes text when it successfully matches it, moving the internal pointer forward so that subsequent parsers only see the remaining unconsumed text.
  8. Best practices for whitespace consumption

    master

    A recommended strategy for managing whitespace is to delay its consumption until the highest possible point in your parser hierarchy. This provides maximum flexibility and makes the role of whitespace explicit in your language definition.

    Additionally, aim to make each individual parser responsible for parsing the smallest possible unit that makes sense for its name.

    const JS = Parsimmon.createLanguage({
      _: () => Parsimmon.regexp(/[ \t]*/), // Optional whitespace
      __: () => Parsimmon.regexp(/[ \t]+/), // Mandatory whitespace
      Var: () => Parsimmon.string("var"),
      "=": () => Parsimmon.string("="),
      Identifier: () => Parsimmon.regexp(/[a-z]+/),
      Definition: r =>
        Parsimmon.seqObj(
          r.Var,
          r.__,
          ["name", r.Identifier],
          r._,
          r["="],
          r._,
          ["value", r.Expression],
          r._,
          r[";"]
        ),
      Expression: () => Parsimmon.fail("TODO: Implement expressions")
    });
  9. Optimize whitespace parsing with RegExp

    master
    When matching large chunks of whitespace, use Parsimmon.regexp instead of character-oriented parsers. Parsimmon.regexp is significantly faster because it avoids examining characters one by one and building arrays.
  10. Parse strings with .parse() and .tryParse()

    master

    Parsimmon parsers represent actions on a text stream. You can execute a parser using two primary methods:

    1. .parse(string): Returns a result object.
      • If successful: { status: true, value: <yielded_value> }.
      • If failed: { status: false, index: <error_index>, expected: [<messages>], error: { offset, line, column } }.
    2. .tryParse(string): Returns the yielded value directly if successful, or throws an error if the parse fails.

    You can use Parsimmon.formatError(source, error) to convert a parse error object into a human-readable string using the original source text.

  11. Build complex languages with Parsimmon.createLanguage(parsers)

    master

    Parsimmon.createLanguage(parsers) is the recommended way to build a full language parser. It organizes parsers into a single namespace and automatically handles recursive definitions, removing the need for manual Parsimmon.lazy calls.

    Each parser function passed in the parsers object receives a single argument: an object representing the entire language (the namespace). You use this object to refer to other rules within your language.

    Example:

    var Lang = Parsimmon.createLanguage({
      Value: function(r) {
        return Parsimmon.alt(r.Number, r.Symbol, r.List);
      },
      Number: function() {
        return Parsimmon.regexp(/[0-9]+/).map(Number);
      },
      Symbol: function() {
        return Parsimmon.regexp(/[a-z]+/);
      },
      List: function(r) {
        return Parsimmon.string("(")
          .then(r.Value.sepBy(r._))
          .skip(Parsimmon.string(")"));
      },
      _: function() {
        return Parsimmon.optWhitespace;
      }
    });
    Lang.Value.tryParse("(list 1 2 foo (list nice 3 56 989 asdasdas))");
  12. Install and use Parsimmon

    master

    Parsimmon can be used in Node.js environments or directly in the browser.

    Node.js

    Install via npm using the package name parsimmon.

    Browser

    Include Parsimmon via a script tag. It exports a global variable called Parsimmon. You can use unpkg to fetch the latest build.

    Note: Parsimmon is currently unmaintained.