Loretta Toolkit

repository·main·Indexed 41 days ago

https://github.com/lorettadevs/loretta

A Roslyn-based C# and (G)Lua toolkit for lexing, parsing, code analysis, transformation, and code generation. It supports various Lua dialects including Standard Lua (5.1-5.4), LuaJIT (2.0, 2.1), Luau (Roblox), Garry's Mod (GLua), and FiveM. Available via NuGet as Loretta.CodeAnalysis.Lua and Loretta.CodeAnalysis.Lua.Experimental, the latter providing experimental features like constant folding and a Lua minifier.

Tokens
8.7K
Snippets
22
Records
37
Agent score
100%

What's inside Loretta

  1. Overview of Loretta toolkit capabilities

    main
    Loretta is a C# (G)Lua toolkit designed for lexing, parsing, code analysis, transformation, and code generation. It is built from scratch based on Roslyn and the official Lua 5.4 syntax, providing support for multiple Lua dialects and extensions.
  2. What is SourceText?

    main

    SourceText is a core abstraction in Loretta.CodeAnalysis.Text designed for efficient code representation.

    Key Responsibilities:

    • Memory Efficiency: Stores code in a format that avoids the Large Object Heap (LOH).
    • Text Manipulation: Allows obtaining specific characters, substrings, and splitting code into TextLines.
    • Mapping: Maps a TextSpan to a LinePositionSpan for accurate error reporting.
    • Versioning/Diffing: Provides methods to obtain file checksums, calculate changes between versions via SourceText.GetTextChanges, or apply changes via SourceText.WithChanges.
  3. Use LuaSyntaxRewriter to modify syntax trees

    main

    The LuaSyntaxRewriter is a fundamental building block for transforming syntax trees. It allows you to replace specific nodes in a tree without manually managing parent-child relationships. You implement a custom rewriter by inheriting from LuaSyntaxRewriter and overriding Visit methods for the specific node types you want to transform (e.g., VisitCompilationUnit or VisitFunctionCallExpression).

    When a node is modified, you typically return the new node using methods like node.Update(...) or by returning a newly constructed node via SyntaxFactory.

    class MyRewriter : LuaSyntaxRewriter
    {
        public override SyntaxNode? VisitSomeNode(SomeNodeSyntax node)
        {
            // Transform the node and return it
            return node.Update(...);
        }
    }
  4. Understand Trivia in Loretta

    main

    Trivia refers to extraneous syntax that does not impact parsing, such as whitespaces, line breaks, comments, and shebangs (#!). Trivia is attached to the preceding or following token:

    • Leading trivia: All trivia located from the first line break after the previous token up to the current token.
    • Trailing trivia: All trivia after a token up to (and including) the first line break.

    You can manipulate trivia using GetTrailingTrivia() and WithTrailingTrivia(SyntaxTriviaList).

  5. How LuaSyntaxWalker works for tree traversal

    main

    A LuaSyntaxWalker is a fundamental building block used to traverse a syntax tree recursively. By inheriting from LuaSyntaxWalker, you can override specific Visit... methods to perform actions when the walker encounters certain node types (e.g., VisitFunctionCallExpression).

    Key implementation details:

    • Depth Control: When calling the base constructor, you can specify SyntaxWalkerDepth.Node to limit traversal to nodes, ignoring everything below them.
    • Recursive Traversal: When overriding a Visit method, you must call base.Visit[MethodName](node) if you want the walker to continue traversing the children of that node. Failing to do so will stop the traversal for that branch.
    internal class MyWalker : LuaSyntaxWalker
    {
        public MyWalker() : base(SyntaxWalkerDepth.Node) { }
    
        public override void VisitFunctionCallExpression(FunctionCallExpressionSyntax node)
        {
            // Do something with the node
            base.VisitFunctionCallExpression(node); // Crucial for continued traversal
        }
    }
  6. Access Lua scoping and variable information

    main

    To perform semantic analysis like retrieving scopes or variables, you must first create a Script instance from your LuaSyntaxTrees.

    Once a Script is created, you can:

    • Get the global scope via Script.RootScope.
    • Retrieve a specific scope using Script.GetScope(SyntaxNode) or Script.FindScope(SyntaxNode, ScopeKind).
    • Retrieve variable information using Script.GetVariable(SyntaxNode).
    • Retrieve labels for goto statements using Script.GetLabel(SyntaxNode) (works on GotoStatementSyntax or GotoLabelStatementSyntax).
  7. Supported Lua dialects and syntax extensions in Loretta

    main

    Loretta provides broad compatibility across several Lua environments and extensions:

    Core Lua Versions

    • Lua 5.1, 5.2, 5.3, and 5.4
    • LuaJIT 2.0 and 2.1

    Dialect Extensions

    • Garry's Mod Lua (glua): Supports C-style comments (// and /* */) and logical operators (&&, ||, !=, !).
    • Luau (Roblox Lua): Partial support for compound assignments (+=, -=, *=, /=, ^=, %=, ..=), if expressions (if a then b else c), and typed Lua syntax.
    • FiveM: Supports parsing of hash string syntax.
    • LuaJIT Compatibility: Accepts identifiers containing characters above 127/0x7F (e.g., emojis).

    continue Keyword Support

    Loretta allows configuring how the continue keyword is handled via different options:

    • No continue support.
    • Roblox style: continue is treated as a contextual keyword.
    • Garry's Mod style: continue is treated as a full-fledged keyword.
  8. Supported Lua dialects and syntax extensions

    main

    Loretta is a toolkit for lexing, parsing, and analyzing Lua-based code. It supports a wide range of dialects including:

    • Standard Lua: 5.1, 5.2, 5.3, and 5.4.
    • LuaJIT: 2.0 and 2.1 (including LuaJIT's specific identifier support for characters above 0x7F).
    • Luau (Roblox): Supports compound assignments (+=, -=, etc.), if expressions, and typed Lua syntax.
    • Garry's Mod (GLua): Supports C-style comments (// and /* */) and specific operators (&&, ||, !=, !).
    • FiveM: Supports FiveM's hash string syntax.

    Continue Keyword Support: You can configure how the continue keyword is handled via options:

    • No continue support.
    • Luau style: continue is treated as a contextual keyword.
    • Garry's Mod style: continue is treated as a full-fledged keyword.
  9. Check if a Lua node can be converted into a local variable

    main

    A Lua syntax node is considered eligible to be turned into a local variable if it is either:

    1. An IdentifierNameSyntax (e.g., print).
    2. A MemberAccessExpressionSyntax whose base expression is also eligible (e.g., math.ceil or a.b.c).

    Complex expressions like (1 + 1).a are not eligible.

    You can implement this check using the IsKind extension method from Loretta.CodeAnalysis.LuaExtensions to verify the SyntaxKind of the node.

    static bool canTurnIntoLocal(SyntaxNode node)
    {
        if (node.IsKind(SyntaxKind.IdentifierName))
            return true;
        else if (node.IsKind(SyntaxKind.MemberAccessExpression))
            return canTurnIntoLocal(((MemberAccessExpressionSyntax) node).Expression;
        else
            return false;
    }
  10. Create Lua nodes with SyntaxFactory

    main

    The SyntaxFactory is a static class used to instantiate all types of Lua syntax nodes, including identifiers, statements, and lists. Because Loretta syntax trees are immutable, all operations that modify a node return a new instance.

    Common factory methods include:

    • SyntaxFactory.IdentifierName(string text): Creates an identifier.
    • SyntaxFactory.LocalVariableDeclarationStatement(...): Creates a local variable declaration.
    • SyntaxFactory.SeparatedList(IEnumerable<T> items): Creates a syntax list where items are separated by specific tokens.
  11. How the Lua Minifier works

    main

    The minifier operates in two main stages:

    1. Renaming

    The renamer iterates through the script and manages variable names using slots:

    • New Variables: When a new variable is encountered, the minifier requests a slot from the ISlotAllocator and converts that slot into a name using the NamingStrategy.
    • Existing Variables: When an existing variable is encountered, it is renamed using the strategy. If it is the last use of that variable, the minifier releases the slot back to the allocator.

    2. Trivia Removal

    This step removes all 'trivia' (comments and whitespace) from the tree, inserting spaces only where they are syntactically necessary.

  12. Use SegmentedArray<T> to avoid Large Object Heap allocations

    main

    The SegmentedArray<T> type is a specialized collection designed as a high-performance alternative to T[]. It provides O(1) reads and writes and maintains storage density/data locality similar to a standard array, but segments the underlying storage to prevent the underlying arrays from being allocated in the Large Object Heap (LOH).

    When to use SegmentedArray<T>:

    • You are currently using T[] and need to maintain its performance characteristics (O(1) access).
    • You do not need to call .AsMemory() or .AsSpan() on the instance, as these operations require the entire collection to be continuous in memory.
    • Application profiling indicates that T[] allocations in the Large Object Heap are causing performance issues.