YaFF (Yet another Flat Format)

YaFF (Yet another Flat Format) is a high-performance C++ serialization library designed to provide a zero-copy wire format for the Protobuf ecosystem. It uses .proto files as the single source of truth but changes the physical data representation to eliminate runtime overhead.

Key Characteristics

• Zero-copy: No parsing or copying is required on read; data can be read directly from memory-mapped files.
• Protobuf Interoperability: Maintains compatibility with the Protobuf ecosystem, allowing you to use existing .proto schemas.
• High Performance: Uses adaptive layout strategies to bring field access performance close to native C++ structs.

Use Cases

Use YaFF if you:

• Want faster reads while staying within the Protobuf ecosystem.
• Need the lowest possible read overhead in performance-critical (hot) paths.
• Require zero-copy benefits (e.g., reading directly from a buffer or memory-mapped file).
• Control both the producer and consumer sides of the data pipeline within a trusted system.

Schema slices allow you to generate multiple independent runtime representations from a single .proto file. This is useful for shipping lightweight buffers to consumers that only need a subset of the data.

To define a slice, you must annotate every message and field along the path from the root to the data using (yaff.proto.message) and (yaff.proto.field) options.

Key Rules:

• slice_name: Defines which slice the option applies to. If omitted, the field/message is included in all slices.
• id: If omitted in a slice, the original Protobuf number is preserved.
• Resolution: Options are resolved slice-first. An option tagged with a specific slice_name takes precedence over an untagged fallback.
• Generation: Slices are selected at generation time using the TAG and NAMESPACE parameters of yaff_generate.

message Embedding {
    option (yaff.proto.message) = {slice_name: "banner_light"};
    option (yaff.proto.message) = {slice_name: "banner_heavy"};

    // Shared across both slices using its original Protobuf field number
    optional uint64 vector_id = 1 [
        (yaff.proto.field) = {slice_name: "banner_light"},
        (yaff.proto.field) = {slice_name: "banner_heavy"}
    ];

    optional uint64 model_version = 2;

    // Included only in the heavy slice, with an overridden YaFF identifier
    optional uint64 computed_time = 3 [
        (yaff.proto.field) = {slice_name: "banner_heavy", id: 2}
    ];
}

YaFF inherits field numbers from your .proto file. These numbers serve as the unique identifiers for fields within a message.

Best Practices

• Uniqueness: Field numbers must be unique within a message and must never be reused once assigned.
• Optimization: Assign the smallest numbers to fields that are populated most frequently to reduce overhead in certain layouts.
• Gaps: While gaps between numbers work, they may add overhead in some layouts. You can reclaim space by reassigning compact IDs.

Retiring Fields

You can retire a field in three ways, which affects available layouts:

1. [deprecated = true]: Keeps the field in the schema and hides it from generated YaFF interfaces. This preserves all layout options.
2. reserved: Removes the field and its type information. This may limit available layouts.
3. Deleting the field: Behaves like reserved. Avoid this to prevent accidental reuse of field numbers.

If you use reserved but need to keep all layout options open, you can explicitly provide the missing type information to YaFF (see the Advanced section in documentation).

The serialized size of a YaFF message is determined by metadata overhead and how unset fields are encoded.

Flat Layout Behavior

In a Flat Layout, setting field N requires writing every field from 0 to N.

• At full density: It is the most compact format (approx. 25% smaller than FlatBuffers for 5-field structures).
• At low density: It carries high overhead because it still stores the space for fields up to the last one set, leading to a 'plateau' in size regardless of how many intermediate fields are unset.

Sparse Layout Behavior

In a Sparse Layout, setting field N writes metadata for fields 0 to N, but does not write values for unset fields.

• At low density: It is much more efficient than Flat Layout. Its lead over FlatBuffers widens as data thins out (reaching ~39% savings at 5% density).
• At high density: It sits between Flat Layout and FlatBuffers.

Key Advantage: Dynamic Layout

Because YaFF supports Dynamic Layouts, you are not forced to commit to a single tradeoff in your schema. The runtime can select the optimal layout (Flat or Sparse) during serialization based on the actual data being written.

Performance characteristics in YaFF vary based on the access pattern of your application:

Local Access (Hot)

• Definition: Many fields are read from a single message (small working set, e.g., ~2 KB).
• Characteristics: Performance is dominated by the format's per-access overhead and the efficiency of the generated code.
• YaFF Advantage: YaFF's Flat Layout is highly optimized for this, performing significantly faster than Protobuf and FlatBuffers because it minimizes the cost of navigating the structure.

Random Access (Cold)

• Definition: A few fields are read from many different messages (large working set, e.g., ~512 MB).
• Characteristics: Performance is dominated by the cost of the initial cache miss (reaching RAM). The layout's ability to keep data within a single cache line or avoid extra metadata lookups is critical.
• YaFF Advantage: YaFF stays close to the raw C++ baseline because its layouts minimize the number of unpredictable memory misses required to reach a field.