Iguana Serialization Engine

repository·master·Indexed 21 days ago

https://github.com/qicosmos/iguana

A high-performance, universal serialization engine for C++ (C++17, C++20, and C++26) that supports JSON, XML, YAML, and Protobuf through a unified API. It leverages compile-time reflection to serialize objects, offering a transition from macro-based metadata (YLT_REFL) in C++17 to attribute-based static reflection in C++26. The library includes a proto_to_struct tool for generating C++ headers from .proto files and a Python script for automatic macro generation.

Tokens
11.7K
Snippets
31
Records
52
Agent score
79%

What's inside Iguana

  1. Overview of Iguana Serialization Engine

    master
    Iguana is a modern, high-performance universal serialization engine for C++ (C++17 and C++20). It uses compile-time reflection to allow developers to serialize objects into various formats like JSON, XML, YAML, and Protobuf using a unified interface. The core idea is that while the display format changes, the underlying metadata remains constant.
  2. Understand the C++26 Reflection Refactor in Iguana

    master

    Iguana has been refactored to use C++26 static reflection, moving away from macro-based registration and structured binding simulations used in C++17/C++20.

    Core Concepts:

    • Struct as Schema: The structure itself acts as the schema. Field traversal uses std::meta::nonstatic_data_members_of and template for.
    • Attribute-based Metadata: Instead of macros, additional semantics (like field names or skipping fields) are attached directly to fields or types using C++26 attributes [[= ...]].
    • Direct Member Access: Deserialization (JSON/XML/YAML) now splices directly to real members using obj.[:member:] instead of using offsets to convert back to field pointers.
    • Private Member Access: Using std::meta::access_context::unchecked(), the engine can now enumerate private data members without requiring special macros like YLT_REFL_PRIVATE.

    Key Benefits:

    • No longer requires types to be aggregates.
    • Removes the need for macro-generated field0...fieldN structured bindings.
    • Simplifies field name extraction and access.
  3. How C++26 Reflection and Dispatch Work in Iguana

    master

    In the C++26 path, Iguana moves away from macro-generated member lists and offset arithmetic to a native reflection model.

    1. Core Reflection (reflect26_core.hpp): Uses std::meta::nonstatic_data_members_of to recursively collect data members from the class and its base classes. It provides utilities like for_each_data_member which uses template for to access fields directly via t.[:member:].
    2. Runtime Dispatch (reflect26_dispatch.hpp): Since JSON/XML keys are runtime strings, Iguana uses a dispatch layer. It performs a linear traversal of the C++26 field list using template for to match runtime keys against field names (or aliases). Once a match is found, it splices the call directly to the member using obj.[:member:].

    This approach allows for accessing even private data members during enumeration and avoids the limitations of aggregate structured binding (such as the 256-field limit).

  4. Understand struct_pb constraints and limitations

    master

    When using struct_pb, be aware of the following technical constraints:

    Protobuf Feature Support

    • Supported: proto3 binary wire format (subset of protobuf-output).
    • Not Supported: proto2 (including required fields), extensions, custom options, services, proto3 JSON mapping, or text format.

    Implementation Requirements

    • Base Implementation: Standard to_pb and from_pb operations do not require your struct to derive from iguana::base_impl. However, if you need dynamic reflection (e.g., creating instances or setting field values by name), your struct must derive from iguana::base_impl.
    • Unknown Fields: To preserve unknown fields during serialization/deserialization, you must explicitly declare them using either a field named pb_unknown_fields_field or by using the C++26 attribute [[= iguana::pb_unknown_fields]].
  5. Define metadata for serialization

    master

    To use Iguana, you must first define metadata for your structures so the engine knows which members to serialize.

    • C++20 and newer: If your compiler supports C++20 (e.g., GCC 11+, Clang 13+, MSVC 2022), you do not need to manually define metadata; the engine uses compile-time reflection.
    • C++17: You must use the YLT_REFL macro to define the metadata for your struct.

    For C++26 compilers with static reflection support, you can also use specific annotations to customize field names, skip fields, or handle XML requirements.

    struct person
    {
        std::string  name;
        int          age;
    };
    #if __cplusplus < 202002L
    YLT_REFL(person, name, age) // define meta data for C++17
    #endif
  6. Handle private members in C++26 reflection

    master

    When using the C++26 reflection path, Iguana enumerates private members of a class by default. If a private field is not explicitly marked with skip_field, it will be included in the serialization and deserialization processes for JSON, XML, YAML, and Protobuf.

    To prevent a private field from being serialized, you must use the skip_field annotation.

    class User {
    public:
      int id;
    
    private:
      // This field will be included in serialization unless marked with skip_field
      std::string token;
    };
  7. Use C++26 Attributes for Schema Metadata

    master

    In the C++26 path, you can provide extra schema information to the serialization engine using attributes. This replaces the old macro-based extension points. Supported attributes include:

    • field_name: Specify a custom name for the field in serialized formats.
    • skip_field: Instruct the engine to ignore this specific field.
    • skip_base: Instruct the engine to skip base classes during reflection.
    • xml_required: Marks a field as required for XML deserialization.
    • pb_field: Provides protobuf-specific field metadata (e.g., field numbers).
  8. Define reflectable objects with YLT_REFL

    master

    To enable serialization/deserialization, you must define your object as reflectable.

    • C++20 (gcc11+, clang13+, msvc2022): No extra macro is required; the compiler handles reflection automatically.
    • Pre-C++20: You must use the YLT_REFL macro to define metadata for the object.

    If your class has private fields, you must place the YLT_REFL macro inside the class definition and ensure it is in the public section.

    struct person
    {
        std::string_view name;
        int age;
    };
    
    #if __cplusplus < 202002L
    YLT_REFL(person, name, age);
    #endif
    
    // For private fields:
    class person {
        std::string name;
        int age;
    public:
        YLT_REFL(person, name, age);
    };
  9. Define Protobuf-compatible structures with struct_pb

    master

    Use struct_pb to define C++ structures that can be serialized/deserialized to the Protobuf format without needing .proto files or protoc.

    1. Define your struct with standard C++ types.
    2. Use the YLT_REFL macro to register the fields for reflection.
    3. Use iguana::to_pb to serialize to a string and iguana::from_pb to deserialize.
    #include <ylt/struct_pb.hpp>
    
    struct my_struct {
      int x;
      bool y;
      struct_pb::fixed64_t z;
    };
    YLT_REFL(my_struct, x, y, z);
    
    struct nest {
      std::string name;
      my_struct value;
      int var;
    };
    YLT_REFL(nest, name, value, var);
    
    int main() {
      nest v{"Hi", {1, false, {3}}, 5}, v2{};
      std::string s;
      iguana::to_pb(v, s);
      iguana::from_pb(v2, s);
      assert(v.var == v2.var);
    }
  10. Run Protobuf conformance tests

    master

    To ensure Iguana's Protobuf implementation is compatible with official standards, you can run the iguana_conformance target against the official Protobuf conformance_test_runner.

    This requires a local build of Protobuf (e.g., version v3.21.12) with protobuf_BUILD_CONFORMANCE=ON enabled.

  11. Customize Protobuf field numbers

    master

    By default, field numbers are assigned sequentially starting from 1 based on declaration order. To interoperate with existing .proto files, you must explicitly specify field numbers.

    Option 1: Using YLT_REFL_PB (Legacy/Non-C++26)

    Use the YLT_REFL_PB macro to map field names to specific numbers.

    Option 2: Using C++26 Attributes

    If using C++26 with reflection enabled, use [[= iguana::pb_field(N)]] attributes directly on the struct members. This is the preferred method for modern C++ environments.

    // Legacy/Non-C++26 path
    struct account {
      std::string name;
      int32_t age;
      std::vector<std::string> emails;
    };
    YLT_REFL_PB(account, (name, 10), (age, 20), (emails, 9));
    
    // C++26 path
    struct event_msg26 {
      [[= iguana::pb_field(1)]] int32_t id{};
      [[= iguana::pb_field(3)]] [[= iguana::pb_bytes]] std::string payload;
    };
  12. Generate C++ struct pack files from .proto files

    master

    Use the protoc compiler with the proto_to_struct plugin to generate C++ header files from your .proto definitions. This allows you to create C++ structs that are compatible with the Iguana serialization engine using the YLT_REFL macro.

    Basic usage: protoc --plugin=protoc-gen-custom=./build/proto_to_struct <input_file>.proto --custom_out=:<output_directory>

    protoc --plugin=protoc-gen-custom=./build/proto_to_struct  data.proto --custom_out=:./protos