CppAst.NET Documentation

repository·main·Indexed 20 days ago

https://github.com/xoofx/cppast.net

A C/C++ parser for .NET that provides a managed Abstract Syntax Tree (AST) model, macro support, and access to comments. Designed as a foundation for domain-oriented PInvoke and Interop code generation, it supports C, C++, and Objective-C via the CppParser class. The library provides a type system for primitive, class, enum, and pointer types, and supports three types of attributes: CxxSystemAttribute, AnnotateAttribute for custom metadata, and legacy TokenAttributes.

Tokens
9.4K
Snippets
20
Records
33
Agent score
67%

What's inside CppAst.NET

  1. Overview of C++ Code Generation via AST

    main

    Modern game engines often rely on code generation to automate tasks like reflection registration. Instead of manual implementation, which is error-prone and labor-intensive, tools can parse C++ source code to extract information and generate registration code automatically.

    A common pattern is the Two-Pass Compilation approach:

    1. First Pass: The tool parses only the header files using libclang to extract essential definitions (e.g., class structures, member variables).
    2. Second Pass: The actual compilation process occurs, including the newly generated registration code alongside the original source.

    This ensures a complete runtime reflection system while maintaining the integrity of the original source code.

  2. Challenges in C++ Type System parsing

    main

    Parsing C++ types is significantly more complex than other languages due to several factors that offline tools must handle:

    • Built-in and User-Defined Types (UDTs): A mix of native types and custom classes/enums.
    • Advanced Type Modifiers: Support for Pointers, References, and Arrays, which can be nested.
    • Type Qualifiers: Types modified by const, volatile, etc.
    • Type Aliasing: The use of using and typedef to create aliases.
    • Modern C++ Keywords: Use of auto, decltype, and typeof for type expressions.
    • Templates: Highly complex template systems for expressing types.

    CppAst.NET addresses these challenges by providing a relatively complete implementation of the C++ type system at the C# layer, allowing developers to analyze types layer-by-layer in a way that closely mirrors native C++ behavior.

  3. Handle C++ Templates and Partial Specialization

    main

    CppAst.NET provides support for identifying and distinguishing between base templates, full specializations, and partial specializations. This allows you to accurately retrieve information about template parameters and their specialized instances.

    Key properties for template analysis:

    • CppTemplateKind.TemplateClass: Represents the base template.
    • CppTemplateKind.TemplateSpecializedClass: Represents a fully specialized class.
    • CppTemplateKind.PartialTemplateClass: Represents a partially specialized template.
    • SpecializedTemplate: Links a specialized class back to its partial template.
    • TemplateSpecializedArguments: A collection of arguments used in the specialization. Use IsSpecializedArgument to determine if a specific argument was part of the specialization pattern or just a standard template parameter.
    // Example of inspecting template specialization via unit test pattern
    var baseTemplate = compilation.Classes[0]; // TemplateClass
    var fullSpecializedClass = compilation.Classes[1]; // TemplateSpecializedClass
    var partialSpecializedTemplate = compilation.Classes[2]; // PartialTemplateClass
    
    // Verify relationship
    Assert.AreEqual(partialSpecializedTemplate, fullSpecializedClass.SpecializedTemplate);
    
    // Check arguments
    var arg = partialSpecializedTemplate.TemplateSpecializedArguments[0];
    Assert.AreEqual("int", arg.ArgString);
  4. Understanding the CppAst.NET Code Generation Workflow

    main

    CppAst.NET is designed to facilitate automated code generation (such as reflection registration or scripting middleware) by extracting information from the C++ Abstract Syntax Tree (AST).

    A common pattern for using CppAst.NET in large-scale engineering (like game engines) is the 2-pass compilation mode:

    1. First Pass (Extraction): Use CppAst.NET (powered by libclang) to process only the header files. This pass extracts necessary metadata, such as class definitions, properties, and methods.
    2. Code Generation: Use the extracted AST information to generate new C++ code (e.g., registration macros or wrapper functions).
    3. Second Pass (Actual Compilation): Perform the standard C++ compilation process, including both the original source files and the newly generated files.

    This approach allows you to build tools that inject additional information into your codebase without manually writing boilerplate registration code.

  5. Understand the three types of C++ attributes in CppAst.NET

    main

    CppAst.NET categorizes attributes into three distinct types to balance performance, compatibility, and functionality. Understanding these is crucial for correctly accessing metadata from parsed C++ code:

    1. AttributeKind.CxxSystemAttribute: High-performance attributes natively supported by libclang (e.g., [[deprecated]], [[noreturn]], visibility). These are always available in the Attributes collection and do not require special configuration.
    2. AttributeKind.AnnotateAttribute: The recommended way to inject custom metadata (meta-attributes) into your code. It uses Clang's annotate mechanism to bypass the limitations and performance costs of token-based parsing. These are stored in the Attributes collection.
    3. AttributeKind.TokenAttribute: Legacy attributes parsed via the tokenizer. These are deprecated and have been moved to a separate TokenAttributes property to distinguish them from the newer, more efficient types. Using these requires enabling a specific parser option.
  6. Configure CppParserOptions for advanced parsing

    main

    CppAst.NET allows fine-grained control over the parsing process via CppParserOptions. Key configuration options include:

    • ParserKind: Supports C, C++, and Objective-C language modes.
    • ParseTokenAttributes: Enables optional token-level attributes.
    • ParseCommentAttribute: Enables comment-based attributes.
    • ParseFunctionBodies: Provides function body source spans (note: this is not a full statement-body AST).
    • ParseMacros: Enables access to macro definitions, parameters, and tokens (default is false).
  7. Handle C++ Templates and Specializations

    main

    CppAst.NET supports full templates, partial specializations, and template specializations. It allows you to distinguish between template parameters and template specialization parameters, enabling precise retrieval of template instance information.

    Key concepts:

    • CppTemplateKind.TemplateClass: A base template.
    • CppTemplateKind.TemplateSpecializedClass: A fully specialized class.
    • CppTemplateKind.PartialTemplateClass: A partially specialized template.
    • SpecializedTemplate: A property on a specialized class that points back to the partial template it was specialized from.
    • TemplateSpecializedArguments: A collection of arguments used in the specialization. You can check IsSpecializedArgument to see if a specific argument was part of the specialization pattern or just a remaining template parameter.
    // Example of asserting template specialization properties
    var baseTemplate = compilation.Classes[0]; // TemplateClass
    var fullSpecializedClass = compilation.Classes[1]; // TemplateSpecializedClass
    var partialSpecializedTemplate = compilation.Classes[2]; // PartialTemplateClass
    
    Assert.AreEqual(baseTemplate.TemplateKind, CppAst.CppTemplateKind.TemplateClass);
    Assert.AreEqual(fullSpecializedClass.TemplateKind, CppAst.CppTemplateKind.TemplateSpecializedClass);
    Assert.AreEqual(partialSpecializedTemplate.TemplateKind, CppAst.CppTemplateKind.PartialTemplateClass);
    
    // Link between specialized class and its partial template
    Assert.AreEqual(fullSpecializedClass.SpecializedTemplate, partialSpecializedTemplate);
    
    // Checking arguments in a partial specialization
    // Argument 0 might be specialized (e.g., 'int'), Argument 1 might not be.
    Assert.AreEqual(partialSpecializedTemplate.TemplateSpecializedArguments[0].IsSpecializedArgument, true);
    Assert.AreEqual(partialSpecializedTemplate.TemplateSpecializedArguments[1].IsSpecializedArgument, false);
  8. Understanding libclang's Cursor mechanism

    main

    libclang represents the C++ Abstract Syntax Tree (AST) using the concept of Cursors (CXCursor). Each cursor represents a specific node in the AST (such as a namespace, class, function, or variable) that corresponds to a piece of source code.

    Key characteristics of the libclang approach:

    • One-to-one mapping: There is a strong relationship between the AST nodes and the source code syntax.
    • Callback-based traversal: libclang primarily accesses a node's children using a callback mechanism (e.g., VisitChildren). This means you cannot easily traverse the tree in a single pass or perform multiple passes without re-traversing or manually caching the data.
    • Complexity: While basic cursors are straightforward, higher complexity is found in Stmt (statements) and Exprs (expressions).

    Because of the callback-based nature, tools that require multiple passes over the AST often need to implement an intermediate data layer to decouple the tool from libclang's native AST.

    // Example of the callback-based traversal pattern used in libclang
    private static void PrintASTByCursor(CXCursor cursor, int level, List<string> saveList)
    {
        bool needPrintChild = true;
        saveList.Add(GetOneCursorDetails(cursor, level, out needPrintChild));
    
        unsafe
        {
            PrintCursorInfo cursorInfo = new PrintCursorInfo();
            cursorInfo.Level = level + 1;
            cursorInfo.SaveList = saveList;
            GCHandle cursorInfoHandle = GCHle.Alloc(cursorInfo);
    
            // Accessing children requires a callback (VisitorForPrint)
            cursor.VisitChildren(VisitorForPrint,
                new CXClientData((IntPtr)cursorInfoHandle));
        }
    }
  9. Understand CppAst.NET limitations and model scope

    main

    CppAst.NET is a lightweight managed model over libclang and does not expose every Clang cursor or statement node. Key architectural considerations include:

    • Function Bodies: When requested via ParseFunctionBodies, these are exposed as source spans rather than a full statement AST.
    • Complex C++ Types: Template, dependent, and unexposed types are represented on a best-effort basis. Advanced constructs may require inspecting CppUnexposedType, display names, or diagnostics.
    • Attributes: Token-level attributes are available for compatibility but are considered obsolete; use system or annotate attributes instead.
    • Type Sizes: Type sizes and built-in aliases (like size_t) follow the configured target triple/ABI.
  10. Explore the CppCompilation data model

    main

    The CppCompilation object acts as a complete, C#-friendly representation of the parsed C++ AST. Instead of navigating a complex tree via callbacks, you can access organized collections of C++ entities through top-level properties.

    Key properties available on CppCompilation include:

    • Namespaces: Namespaces contained within the compilation unit.
    • Enums: Enumerations found in the unit.
    • Functions: Global functions found in the unit.
    • Classes: Classes and structs found in the unit.
    • Typedefs: Typedef types found in the unit.
    • Diagnostics.Messages: Diagnostic/error messages produced during parsing.
  11. Understanding the limitations of libclang's Cursor mechanism

    main

    While libclang is the industry standard for converting C++ source code into an Abstract Syntax Tree (AST), it presents several challenges for developers building code generation tools:

    • Callback-based Traversal: libclang primarily uses a CXCursor mechanism where you access child nodes via callbacks (e.g., VisitChildren). This makes it difficult to perform complex, multi-pass analysis that requires repeated access to the same nodes or their children.
    • Complexity of C++ Types: The C++ type system is extremely complex, involving built-in types, User-Defined Types (UDTs) like class and enum, qualifiers (const, volatile), aliases (using, typedef), and advanced types like pointers, references, and arrays.
    • Integration Overhead: Using low-level wrappers (like ClangSharp) often requires developers to manually implement a secondary data layer in their host language (e.g., C#) to decouple the tool from the raw libclang AST and enable easier traversal.
  12. Understand the CppAst type system

    main

    All type classes in CppAst inherit from CppType. The following specific types are available:

    • CppPrimitiveType: Primitive types like int, char, unsigned int.
    • CppClass: Structs, classes, and unions. Use CppClass.ClassKind to distinguish them. Includes template metadata via TemplateKind, TemplateParameters, etc.
    • CppEnum: Scoped and regular enums.
    • CppTypedef: Typedef declarations.
    • CppPointerType: Pointer types (e.g., int*).
    • CppReferenceType: Reference types (e.g., int&).
    • CppArrayType: Array types (e.g., int[5]).
    • CppQualifiedType: Qualified types (e.g., const int).
    • CppFunctionType: Function types (e.g., void (*)(int)).
    • CppBlockFunctionType: Block function types.
    • CppUnexposedType: Clang types that do not have a specific CppAst model.