Superpower Parser Combinator Library

Token-driven parsing provides more precise error messages (e.g., reporting an unexpected identifier instead of an unexpected character). It involves two distinct steps:

1. Tokenization: Using a class derived from Tokenizer<TKind> to split input into a list of tagged tokens (usually defined by an enum).
2. Parsing: Using a TokenListParser<TKind> to consume the list of tokens.

This approach allows errors to be reported in terms of high-level tokens rather than individual characters.

var expression = "1 * (2 + 3)";

// 1. Tokenization
var tokenizer = new ArithmeticExpressionTokenizer();
var tokenList = tokenizer.Tokenize(expression);

// 2. Parsing
var parser = ArithmeticExpressionParser.Lambda;
var expressionTree = parser.Parse(tokenList);

Superpower is embedded directly into your C# program. You can add it to your project using the dotnet CLI.

dotnet add package Superpower

Superpower is designed for high performance. To maximize throughput:

1. Use handwritten tokenizers for maximum control and performance.
2. Avoid LINQ comprehensions in parsers; instead, use chained combinators like Then() and IgnoreThen(). These allocate fewer delegates (closures) during the parsing process.

The simplest text parsers consume characters directly from the source text. You can use built-in parsers like Character.EqualTo() and combinators like AtLeastOnce() to build complex logic. You can also use LINQ-style syntax to compose parsers declaratively.

// Parse any number of capital 'A's in a row
var parseA = Character.EqualTo('A').AtLeastOnce();

// Example of a more complex identifier parser using LINQ syntax
TextParser<string> identifier =
    from first in Character.Letter
    from rest in Character.LetterOrDigit.Or(Character.EqualTo('_')).Many()
    select first + new string(rest);

var id = identifier.Parse("abc123");

When using TokenizerBuilder<TKind>.Match, the requireDelimiters parameter controls how the tokenizer handles boundaries.

• Set requireDelimiters: true for keywords or reserved words. This ensures that a keyword like null is not incorrectly matched when it is actually part of a larger identifier like nullability.
• Set requireDelimiters: false (default) for tokens that do not need boundary checks, such as operators or symbols.

The tokenizer handles this by looking ahead to see if the match is followed by a delimiter (an ignored character or end-of-input). If a match is found but the delimiter requirement isn't met, the tokenizer discards that match and continues searching subsequent recognizers.

The DateTimeTextParser sample demonstrates how to use Superpower to build a text parser for ISO-8601 formatted date and time values. This specific implementation covers the following formats:

• YYYY-MM-DD (e.g., 2017-01-01)
• YYYY-MM-DD HH:mm (e.g., 2017-01-01 12:10)
• YYYY-MM-DD HH:mm:ss (e.g., 2017-01-01 12:10:30)

Note that this sample does not cover time zones or fractional seconds.

Token list parsers are defined using TokenListParser<TKind, TResult>. They use the Token class to match specific token types. You can use combinators like .Value(), .Apply(), .Select(), and .Or() to build complex logic, and Parse.Chain() for handling operator precedence and associativity.

// Example snippet of a token parser component
static readonly TokenListParser<ArithmeticExpressionToken, Expression> Constant =
        Token.EqualTo(ArithmeticExpressionToken.Number)
        .Apply(Numerics.IntegerInt32)
        .Select(n => (Expression)Expression.Constant(n));

The TokenizerBuilder<TKind> class allows you to quickly assemble tokenizers from recognizers (text parsers). You can define which patterns to Match to specific token types and which patterns to Ignore (like whitespace). Tokenizers match patterns in top-to-bottom order.

var tokenizer = new TokenizerBuilder<ArithmeticExpressionToken>()
    .Ignore(Span.WhiteSpace)
    .Match(Character.EqualTo('+'), ArithmeticExpressionToken.Plus)
    .Match(Character.EqualTo('-'), ArithmeticExpressionToken.Minus)
    .Match(Character.EqualTo('*'), ArithmeticExpressionToken.Times)
    .Match(Character.EqualTo('/'), ArithmeticExpressionToken.Divide)
    .Match(Character.EqualTo('('), ArithmeticExpressionToken.LParen)
    .Match(Character.EqualTo(')'), ArithmeticExpressionToken.RParen)
    .Match(Numerics.Natural, ArithmeticExpressionToken.Number)
    .Build();

To provide more informative error messages for specific token types, apply the [Token] attribute to your token enum members. You can specify a Category and an Example to help the user understand what was expected.

public enum ArithmeticExpressionToken
{
    None,
    Number,

    [Token(Category = "operator", Example = "+")]
    Plus,
}