Nerdbank.MessagePack

Nerdbank.MessagePack is a high-performance MessagePack serialization library for .NET designed with several key developer-centric features:

• Contractless by default: No attributes are required on data types to enable serialization, though they can be used for customization.
• Modern C# Support: Fully supports required and init properties, record classes/structs, and primary constructors.
• Advanced Deserialization:
  • Polymorphic deserialization: Supports derived types, including duck-typed polymorphic serialization.
  • Targeted deserialization: Use LINQ expressions to deserialize only specific fragments of data.
  • Streaming: Supports true async and endless streaming deserialization for large or continuous data sequences.
• Data Integrity & Evolution:
  • Forward compatibility: Retain unrecognized data during a deserialize/re-serialize cycle.
  • Reference preservation: Optionally preserve reference equality and handle cyclical references.
  • Structural equality: Provides deep by-value equality and hashing for arbitrary types.
• Runtime Compatibility: Optimized for NativeAOT, trimmed environments, SignalR, ASP.NET Core MVC, and Unity.
• Security: Built with secure defaults to protect against untrusted data, including rejection of multiple values for the same property.

Nerdbank.MessagePack is a high-performance MessagePack serialization library designed for modern .NET development. Key capabilities include:

• NativeAOT & Trim Friendly: Works seamlessly in NativeAOT, trimmed, and Unity environments.
• Minimal Configuration: No attributes are required on data types by default, though [GenerateShape] is recommended for performance.
• Modern C# Support: Supports record classes/structs, required and init properties, and primary constructors.
• Advanced Serialization: Supports polymorphic deserialization (unions), streaming deserialization, and targeted deserialization using LINQ expressions.
• Data Integrity: Features forward-compatible data retention (retaining unrecognized properties) and optional preservation of reference equality.
• Safety: Designed with security for untrusted data and contains no mutable statics, making it safe for multi-tenant or complex process environments.

The behavior of a custom converter when applied to a union type depends on where the attribute is applied:

Applied to the Base Type

If the converter is applied to the base type (via MessagePackConverterAttribute on the type or via runtime registration), it will only serialize and deserialize instances of that exact type. The value is automatically enveloped in a msgpack union structure with the appropriate type metadata. The converter does not need to handle derived types or metadata manually.

Note: A custom converter applied to an abstract or interface type will never run because objects can only be instances of concrete types.

Applied to a Member or Parameter

If the converter is applied to a specific member or parameter, it takes total control over the serialization of that value. In this scenario, the custom converter is responsible for handling any union type metadata itself.

The choice between synchronous and asynchronous APIs involves a trade-off between raw speed and memory pressure.

Synchronous (de)serialization

• Performance: Fastest available APIs.
• Allocations: Minimal memory allocations. The library strives to be allocation-free.
• Note: While Serialize<T>(T value, CancellationToken ct) allocates a byte[] for the return value, you can avoid this by using overloads that allow serializing directly to pooled buffers.

Asynchronous (de)serialization

• Performance: Slower than synchronous APIs (ranging from slightly to dramatically slower).
• Memory Pressure: Reduces total memory pressure because the entire serialized representation does not necessarily need to be in memory at once.
• Caveat: Memory pressure improvements are not guaranteed for very large atomic values (like long strings or large byte[] buffers), as these must still be fully in memory during the operation.

When implementing custom converters by deriving from Nerdbank.MessagePack.MessagePackConverter<T>, you must ensure that the Read and Write methods handle exactly one MessagePack structure.

A single structure can be a scalar value (like an integer) or a collection (like an array or a map).

• If writing nothing: You may write an empty array header.
• If writing multiple values: You must lead with an array or map header that exactly predicts the number of values that will follow.
• If reading multiple values: You must first read the header (e.g., ReadArrayHeader) and then process the elements. To maintain compatibility, you should either deserialize exactly the expected number of elements or use a loop to consume all elements in the structure (using reader.Skip() for unknown elements) to ensure the reader position is correctly advanced.

public class MyTypeConverter : MessagePackConverter<MyType>
{
    public override MyType Read(ref MessagePackReader reader, SerializationContext context)
    {
        int count = reader.ReadArrayHeader();
        if (count != 2)
        {
            throw new MessagePackSerializationException("Expected 2 elements.");
        }

        int property1 = reader.ReadInt32();
        short property2 = reader.ReadInt16();
        return new MyType(property1, property2);
    }

    public override void Write(ref MessagePackWriter writer, in MyType value, SerializationContext context)
    {
        writer.WriteArrayHeader(2);
        writer.Write(value.Property1);
        writer.Write(value.Property2);
    }
}

When a type cannot be automatically serialized (e.g., due to private fields or incompatible types), you can define a surrogate type and a marshaler.

• Surrogate Type: A serializable representation of your original type. It is recommended to use a record struct to minimize allocation costs. The surrogate must have at least internal visibility.
• Marshaler: A class implementing IMarshaler<TOriginal, TSurrogate> that handles the conversion between the original type and the surrogate. The marshaler must have at least internal visibility.

Important Note on Nullability: If the original type is a reference type and the surrogate is a value type, specify a nullable surrogate type in the marshaler to ensure null identity is preserved correctly.

// 1. The original unserializable type
[TypeShape(Marshaler = typeof(MyMarshaler))]
public class MyType
{
    private int _secret;
    public MyType(int secret) => _secret = secret;
}

// 2. The surrogate (use a struct for performance)
internal record struct MyTypeSurrogate(int Secret);

// 3. The marshaler
internal class MyMarshaler : IMarshaler<MyType, MyTypeSurrogate>
{
    public MyTypeSurrogate ToSurrogate(MyType original) 
        => new MyTypeSurrogate(original._secret);

    public MyType FromSurrogate(MyTypeSurrogate surrogate) 
        => new MyType(surrogate.Secret);
}