Iguana Serialization Engine
repository·master·Indexed 21 days ago
https://github.com/qicosmos/iguanaA high-performance, universal serialization engine for C++ (C++17, C++20, and C++26) that supports JSON, XML, YAML, and Protobuf through a unified API. It leverages compile-time reflection to serialize objects, offering a transition from macro-based metadata (YLT_REFL) in C++17 to attribute-based static reflection in C++26. The library includes a proto_to_struct tool for generating C++ headers from .proto files and a Python script for automatic macro generation.
What's inside Iguana
- Iguana is a modern, high-performance universal serialization engine for C++ (C++17 and C++20). It uses compile-time reflection to allow developers to serialize objects into various formats like JSON, XML, YAML, and Protobuf using a unified interface. The core idea is that while the display format changes, the underlying metadata remains constant.
Understand the C++26 Reflection Refactor in Iguana
masterIguana has been refactored to use C++26 static reflection, moving away from macro-based registration and structured binding simulations used in C++17/C++20.
Core Concepts:
- Struct as Schema: The structure itself acts as the schema. Field traversal uses
std::meta::nonstatic_data_members_ofandtemplate for. - Attribute-based Metadata: Instead of macros, additional semantics (like field names or skipping fields) are attached directly to fields or types using C++26 attributes
[[= ...]]. - Direct Member Access: Deserialization (JSON/XML/YAML) now splices directly to real members using
obj.[:member:]instead of using offsets to convert back to field pointers. - Private Member Access: Using
std::meta::access_context::unchecked(), the engine can now enumerate private data members without requiring special macros likeYLT_REFL_PRIVATE.
Key Benefits:
- No longer requires types to be aggregates.
- Removes the need for macro-generated
field0...fieldNstructured bindings. - Simplifies field name extraction and access.
- Struct as Schema: The structure itself acts as the schema. Field traversal uses
How C++26 Reflection and Dispatch Work in Iguana
masterIn the C++26 path, Iguana moves away from macro-generated member lists and offset arithmetic to a native reflection model.
- Core Reflection (
reflect26_core.hpp): Usesstd::meta::nonstatic_data_members_ofto recursively collect data members from the class and its base classes. It provides utilities likefor_each_data_memberwhich usestemplate forto access fields directly viat.[:member:]. - Runtime Dispatch (
reflect26_dispatch.hpp): Since JSON/XML keys are runtime strings, Iguana uses a dispatch layer. It performs a linear traversal of the C++26 field list usingtemplate forto match runtime keys against field names (or aliases). Once a match is found, it splices the call directly to the member usingobj.[:member:].
This approach allows for accessing even
privatedata members during enumeration and avoids the limitations of aggregate structured binding (such as the 256-field limit).- Core Reflection (
Understand struct_pb constraints and limitations
masterWhen using
struct_pb, be aware of the following technical constraints:Protobuf Feature Support
- Supported:
proto3binary wire format (subset ofprotobuf-output). - Not Supported:
proto2(includingrequiredfields),extensions,custom options,services,proto3 JSON mapping, ortext format.
Implementation Requirements
- Base Implementation: Standard
to_pbandfrom_pboperations do not require your struct to derive fromiguana::base_impl. However, if you need dynamic reflection (e.g., creating instances or setting field values by name), your struct must derive fromiguana::base_impl. - Unknown Fields: To preserve unknown fields during serialization/deserialization, you must explicitly declare them using either a field named
pb_unknown_fields_fieldor by using the C++26 attribute[[= iguana::pb_unknown_fields]].
- Supported:
Define metadata for serialization
masterTo use Iguana, you must first define metadata for your structures so the engine knows which members to serialize.
- C++20 and newer: If your compiler supports C++20 (e.g., GCC 11+, Clang 13+, MSVC 2022), you do not need to manually define metadata; the engine uses compile-time reflection.
- C++17: You must use the
YLT_REFLmacro to define the metadata for your struct.
For C++26 compilers with static reflection support, you can also use specific annotations to customize field names, skip fields, or handle XML requirements.
struct person { std::string name; int age; }; #if __cplusplus < 202002L YLT_REFL(person, name, age) // define meta data for C++17 #endifHandle private members in C++26 reflection
masterWhen using the C++26 reflection path, Iguana enumerates private members of a class by default. If a private field is not explicitly marked with
skip_field, it will be included in the serialization and deserialization processes for JSON, XML, YAML, and Protobuf.To prevent a private field from being serialized, you must use the
skip_fieldannotation.class User { public: int id; private: // This field will be included in serialization unless marked with skip_field std::string token; };Use C++26 Attributes for Schema Metadata
masterIn the C++26 path, you can provide extra schema information to the serialization engine using attributes. This replaces the old macro-based extension points. Supported attributes include:
field_name: Specify a custom name for the field in serialized formats.skip_field: Instruct the engine to ignore this specific field.skip_base: Instruct the engine to skip base classes during reflection.xml_required: Marks a field as required for XML deserialization.pb_field: Provides protobuf-specific field metadata (e.g., field numbers).
Define reflectable objects with YLT_REFL
masterTo enable serialization/deserialization, you must define your object as reflectable.
- C++20 (gcc11+, clang13+, msvc2022): No extra macro is required; the compiler handles reflection automatically.
- Pre-C++20: You must use the
YLT_REFLmacro to define metadata for the object.
If your class has private fields, you must place the
YLT_REFLmacro inside the class definition and ensure it is in thepublicsection.struct person { std::string_view name; int age; }; #if __cplusplus < 202002L YLT_REFL(person, name, age); #endif // For private fields: class person { std::string name; int age; public: YLT_REFL(person, name, age); };Define Protobuf-compatible structures with struct_pb
masterUse
struct_pbto define C++ structures that can be serialized/deserialized to the Protobuf format without needing.protofiles orprotoc.- Define your struct with standard C++ types.
- Use the
YLT_REFLmacro to register the fields for reflection. - Use
iguana::to_pbto serialize to a string andiguana::from_pbto deserialize.
#include <ylt/struct_pb.hpp> struct my_struct { int x; bool y; struct_pb::fixed64_t z; }; YLT_REFL(my_struct, x, y, z); struct nest { std::string name; my_struct value; int var; }; YLT_REFL(nest, name, value, var); int main() { nest v{"Hi", {1, false, {3}}, 5}, v2{}; std::string s; iguana::to_pb(v, s); iguana::from_pb(v2, s); assert(v.var == v2.var); }Run Protobuf conformance tests
masterTo ensure Iguana's Protobuf implementation is compatible with official standards, you can run the
iguana_conformancetarget against the official Protobufconformance_test_runner.This requires a local build of Protobuf (e.g., version
v3.21.12) withprotobuf_BUILD_CONFORMANCE=ONenabled.Customize Protobuf field numbers
masterBy default, field numbers are assigned sequentially starting from 1 based on declaration order. To interoperate with existing
.protofiles, you must explicitly specify field numbers.Option 1: Using
YLT_REFL_PB(Legacy/Non-C++26)Use the
YLT_REFL_PBmacro to map field names to specific numbers.Option 2: Using C++26 Attributes
If using C++26 with reflection enabled, use
[[= iguana::pb_field(N)]]attributes directly on the struct members. This is the preferred method for modern C++ environments.// Legacy/Non-C++26 path struct account { std::string name; int32_t age; std::vector<std::string> emails; }; YLT_REFL_PB(account, (name, 10), (age, 20), (emails, 9)); // C++26 path struct event_msg26 { [[= iguana::pb_field(1)]] int32_t id{}; [[= iguana::pb_field(3)]] [[= iguana::pb_bytes]] std::string payload; };Generate C++ struct pack files from .proto files
masterUse the
protoccompiler with theproto_to_structplugin to generate C++ header files from your.protodefinitions. This allows you to create C++ structs that are compatible with the Iguana serialization engine using theYLT_REFLmacro.Basic usage:
protoc --plugin=protoc-gen-custom=./build/proto_to_struct <input_file>.proto --custom_out=:<output_directory>protoc --plugin=protoc-gen-custom=./build/proto_to_struct data.proto --custom_out=:./protos