CSLY (C# Lex Yacc)

repository·dev·Indexed 19 days ago

A parser generator for C# that allows developers to define lexers and parsers within a single class using attributes and BNF/EBNF notation. It provides a middle ground between parser combinators and heavy-duty tools like ANTLR, supporting both flexible Regex-based lexers and high-performance Generic lexers. Features include support for LL recursive descent and EBNF multipliers (* and +) via ParserBuilder.

Tokens: 3.6K
Snippets: 7
Records: 9
Agent score: 15%

What's inside CSLY

Use EBNF notation for repeated elements
dev
When using ParserType.EBNF_LL_RECURSIVE_DESCENT, you can use EBNF multipliers in your [Production] strings:
- *: Zero or more repetitions.
- +: One or more repetitions.
Method Parameter Types for EBNF: When a rule involves repeated elements, the corresponding method parameters must be:
- List<TOut> for repeated non-terminals.
- List<Token<TIn>> for repeated terminals.
https://github.com/b3b00/csly/blob/dev/index.md
Choose between Regex-based and Generic lexers
dev
CSLY provides two lexer implementations depending on your needs for flexibility versus performance:
- Regex-based Lexer: Highly flexible as it uses regular expressions to define lexemes. However, it is slower and can become a performance bottleneck.
- Generic Lexer: Offers significantly better performance by using a limited set of predefined token types. The trade-off is reduced flexibility (e.g., identifiers are restricted to alpha characters).
To use either, you must define a C# enum where each member represents a token and is decorated with the [Lexeme] attribute.
https://github.com/b3b00/csly/blob/dev/lexer.md
Define grammar rules using [Production] attributes
dev
A parser is defined within a single class that contains both the lexer configuration and the grammar rules. Grammar rules are mapped to methods using the [Production("rule name")] attribute following BNF notation.
Method Signature Rules:
- Terminals: A method parameter for a terminal must be of type Token<T>, where T is your token enum.
- Non-terminals: A method parameter for a non-terminal must be of type TOut (the parser's output type), representing the result of that rule's evaluation.
- Case Sensitivity: Terminal names in the production string must exactly match the enum member names (case-sensitive).
Example of a rule mapping: [Production("expression : term PLUS expression")] maps to a method Expression(int left, Token<ExpressionToken> operatorToken, int right).
```
[Production("primary: INT")]
public int Primary(Token<ExpressionToken> intToken)
{
    return intToken.IntValue;
}

[Production("expression : term PLUS expression")]
[Production("expression : term MINUS expression")]
public int Expression(int left, Token<ExpressionToken> operatorToken, int right)
{
    // logic to return result based on operatorToken.TokenID
}
```
https://github.com/b3b00/csly/blob/dev/index.md
Configure the Lexer using an Enum and [Lexeme] attributes
dev
The lexer is configured by defining a public enum where each member represents a token. Each enum value must be decorated with the [Lexeme] attribute to define its regular expression.
[Lexeme] parameters:
- string regex: The regular expression for the token.
- boolean isSkippable (optional, default false): If true, the lexer ignores this token (e.g., whitespace).
- boolean isLineending (optional, default false): If true, allows for line counting during lexing.
You can use the lexer independently of the parser using Lexer.Tokenize<T>(source) or by building a specific lexer instance with LexerBuilder.BuildLexer<T>().
```
public enum ExpressionToken
{
    [Lexeme("[0-9]+\\.[0-9]+")]
    DOUBLE = 1,

    [Lexeme("[0-9]+")]
    INT = 3,

    [Lexeme("\\+")]
    PLUS = 4,

    [Lexeme("[ \t]+", true)]
    WS = 12, 

    [Lexeme("[\\n\\r]+", true, true)]
    EOL = 14
}

// Usage: Independent Lexing
ILexer<ExpressionToken> lexer = LexerBuilder.BuildLexer<ExpressionToken>();
var tokens = lexer.Tokenize(source).ToList();
```
https://github.com/b3b00/csly/blob/dev/index.md

Configure a Regex-based lexer

dev

To use the regex-based lexer, decorate your token enum members with the [Lexeme] attribute using a regular expression string.

Supported parameters for [Lexeme(regex, isSkippable, isLineending)]:

string regex: The regular expression that captures the lexeme.
boolean isSkippable (optional, default false): If true, the lexer ignores this token (useful for whitespace).
boolean isLineending (optional, default false): If true, the lexeme matches a line end, allowing for accurate line counting during lexing.

Example configuration for a mathematical parser:

public enum ExpressionToken
{
    [Lexeme("[0-9]+\\.[0-9]+")]
    DOUBLE = 1,

    [Lexeme("[0-9]+")]
    INT = 3,

    [Lexeme("\\+")]
    PLUS = 4,

    [Lexeme("[ \t]+", true)]
    WS = 12, 

    [Lexeme("[\\n\\r]+", true, true)]
    EOL = 14
}

public enum ExpressionToken
{
    // float number 
    [Lexeme("[0-9]+\\.[0-9]+")]
    DOUBLE = 1,

    // integer        
    [Lexeme("[0-9]+")]
    INT = 3,

    // the + operator
    [Lexeme("\\+")]
    PLUS = 4,

    // the - operator
    [Lexeme("\\-")]
    MINUS = 5,

    // the * operator
    [Lexeme("\\*")]
    TIMES = 6,

    //  the  / operator
    [Lexeme("\\/")]
    DIVIDE = 7,

    // a left paranthesis (
    [Lexeme("\\(")]
    LPAREN = 8,

    // a right paranthesis )
    [Lexeme("\\)")]
    RPAREN = 9,

    // a whitespace
    [Lexeme("[ \t]+",true)]
    WS = 12, 

    [Lexeme("[\\n\\r]+", true, true)]
    EOL = 14
}

Install SLY via NuGet or dotnet CLI
dev
To use SLY in your C# project, install it using the NuGet Package Manager or the dotnet CLI.
```
# Using Package Manager Console
Install-Package sly

# Using dotnet CLI
dotnet add package sly
```
https://github.com/b3b00/csly/blob/dev/index.md

Configure a Generic lexer

dev

The generic lexer uses a predefined set of GenericToken types to achieve high performance. You map your custom enum tokens to these generic types using the [Lexeme] attribute.

Static Lexemes

These allow a 1-to-1 mapping and only require the GenericToken as a parameter:

GenericToken.Identifier: Matches alpha characters (A-Z, a-z).
GenericToken.String: Matches strings delimited by double quotes.
GenericToken.Int: Matches a series of digits.
GenericToken.Double: Matches a float (using . as decimal separator).

Example: [Lexeme(GenericToken.String)]

Configurable Lexemes

These require two parameters: the GenericToken and the specific value (string) to match:

GenericToken.keyWord: An identifier with special meaning.
GenericToken.SugarToken: A general-purpose lexeme (must start with an alpha character).

Example: [Lexeme(GenericToken.KeyWord, "if")] or [Lexeme(GenericToken.SugarToken, ">")]

    {
        #region keywords 0 -> 19

        [Lexeme(GenericToken.KeyWord,"if")]
        IF = 1,

        [Lexeme(GenericToken.KeyWord, "then")]
        THEN = 2,

        [Lexeme(GenericToken.KeyWord, "else")]
        ELSE = 3,

        [Lexeme(GenericToken.KeyWord, "while")]
        WHILE = 4,

        [Lexeme(GenericToken.KeyWord, "do")]
        DO = 5,

        [Lexeme(GenericToken.KeyWord, "skip")]
        SKIP = 6,

        [Lexeme(GenericToken.KeyWord, "true")]
        TRUE = 7,

        [Lexeme(GenericToken.KeyWord, "false")]
        FALSE = 8,
        [Lexeme(GenericToken.KeyWord, "not")]
        NOT = 9,

        [Lexeme(GenericToken.KeyWord, "and")]
        AND = 10,

        [Lexeme(GenericToken.KeyWord, "or")]
        OR = 11,

        [Lexeme(GenericToken.KeyWord, "(print)")]
        PRINT = 12,

        #endregion

        #region literals 20 -> 29

        [Lexeme(GenericToken.Identifier)]
        IDENTIFIER = 20,

        [Lexeme(GenericToken.String)]
        STRING = 21,

        [Lexeme(GenericToken.Int)]
        INT = 22,

        #endregion

        #region operators 30 -> 49

        [Lexeme(GenericToken.SugarToken,">")]
        GREATER = 30,

        [Lexeme(GenericToken.SugarToken, "<")]
        LESSER = 31,

        [Lexeme(GenericToken.SugarToken, "==")]
        EQUALS = 32,

        [Lexeme(GenericToken.SugarToken, "!=")]
        DIFFERENT = 33,

        [Lexeme(GenericToken.SugarToken, ".")]
        CONCAT = 34,

        [Lexeme(GenericToken.SugarToken, ":=")]
        ASSIGN = 35,

        [Lexeme(GenericToken.SugarToken, "+")]
        PLUS = 36,

        [Lexeme(GenericToken.SugarToken, "-")]
        MINUS = 37,


        [Lexeme(GenericToken.SugarToken, "*")]
        TIMES = 38,

        [Lexeme(GenericToken.SugarToken, "/")]
        DIVIDE = 39,

        #endregion 

        #region sugar 50 ->

        [Lexeme(GenericToken.SugarToken, "(")]
        LPAREN = 50,

        [Lexeme(GenericToken.SugarToken, ")")]
        RPAREN = 51,

        [Lexeme(GenericToken.SugarToken, ";")]
        SEMICOLON = 52,
    


        EOF = 0

        #endregion

    }

Build and use a Parser with ParserBuilder
dev
To create a functional parser, pass an instance of your definition class to ParserBuilder.BuildParser<TIn, TOut>.
Parameters:
1. definition: An instance of your class containing [LexerConfiguration] and [Production] attributes.
2. type: The ParserType:
  ParserType.LL_RECURSIVE_DESCENT: For standard BNF grammars.
  ParserType.EBNF_LL_RECURSIVE_DESCENT: For EBNF grammars (supports * and + operators).
3. rootRule: A string representing the starting rule of your grammar.
Parsing Results: Calling .Parse(string content) returns a ParseResult<TIn>. You should check r.IsError before accessing r.Result. If errors exist, they are available in r.Errors.
```
ExpressionParser definition = new ExpressionParser();

// 2. Build the parser
Parser<ExpressionToken, int> parser = ParserBuilder.BuildParser<ExpressionToken, int>(
    definition, 
    ParserType.LL_RECURSIVE_DESCENT, 
    "expression"
);

// 3. Execute
string expression = "1 + 1";
ParseResult<ExpressionToken> r = parser.Parse(expression);

if (!r.IsError && r.Result is int result)
{
    Console.WriteLine($"Result: {result}");
}
else if (r.Errors != null)
{
    r.Errors.ForEach(e => Console.WriteLine(e.ErrorMessage));
}
```
https://github.com/b3b00/csly/blob/dev/index.md
Tokenize source code using the Lexer
dev
You can tokenize a string using the static Lexer.Tokenize method, which returns an IEnumerable<Token<T>>.
Alternatively, you can build a specific lexer instance using LexerBuilder.BuildLexer<T>() and then call .Tokenize(source) on that instance.
```
IList<Token<T>> tokens = Lexer.Tokenize(source).ToList<Token<T>>();

// Building and using a specific lexer instance
ILexer<ExpressionToken> lexer = LexerBuilder.BuildLexer<ExpressionToken>();
var tokens = lexer.Tokenize(source).ToList();
```
https://github.com/b3b00/csly/blob/dev/lexer.md