DOCSEARCHbyAlgolia
Submit Docs

FlatBuffers Serialization Library

repository·master·Indexed 12 days ago

https://github.com/google/flatbuffers

A cross-platform memory-efficient serialization library (version 25.12.19) that enables direct access to serialized data without intermediate parsing or unpacking. It includes the flatc compiler for generating language-specific classes from .fbs schemas and provides support for various platforms including C++, Go, Dart, Swift, Python, and TypeScript, with integrated gRPC support.

Tokens
51.9K
Snippets
171
Records
251
Agent score
98%

What's inside FlatBuffers

  1. Overview of FlatBuffers

    master
    FlatBuffers is an efficient, cross-platform serialization library designed for performance-critical applications like game development. It allows for direct access to serialized data without the need for a parsing or unpacking step, which minimizes memory overhead and increases speed. It is available under the Apache license v2.0 and supports a wide range of languages including C++, C#, C, Go, Java, Kotlin, JavaScript, Lobster, Lua, TypeScript, PHP, Python, Rust, and Swift.

  2. Supported Operating Systems and Languages

    master

    FlatBuffers supports a wide range of environments and programming languages.

    Supported Operating Systems:

    • Windows
    • macOS
    • Linux
    • Android
    • Any system with a recent C++ compiler (C++ 11 and newer)

    Supported Programming Languages:

    • C, C++, C#
    • Dart, Go, Java, JavaScript, Kotlin, Lobster, Lua, PHP, Python, Rust, Swift, TypeScript, Nim

  3. Use FlatBuffers with Dart

    master

    To use FlatBuffers in a Dart project, you must use the flatc (FlatBuffer compiler) binary to generate Dart classes from a FlatBuffers IDL schema. These generated classes allow you to read and write binary data that is interoperable with other languages and platforms supported by FlatBuffers.

    Workflow:

    1. Define your data structure in a .fbs schema file.
    2. Use the flatc compiler to generate Dart code from that schema.
    3. Use the generated Dart classes in your application to serialize or deserialize data.

  4. What is FlexBuffers and when to use it

    master

    FlexBuffers is a schema-less binary serialization format designed for storing data that doesn't fit a predefined schema. While regular FlatBuffers require a schema for maximum performance and strong typing, FlexBuffers allows for free-form data structures.

    Key Characteristics:

    • Schema-less: You can store data without knowing the structure ahead of time.
    • Zero-copy access: Like regular FlatBuffers, FlexBuffers can be accessed without parsing, copying, or object allocation, making it highly efficient for mmap-ing large amounts of data.
    • Compact encoding: It uses automatic pooling of strings and automatic sizing of containers (8/16/32/64 bits) to minimize binary size.
    • Performance Trade-off: FlexBuffers is slower than regular schema-based FlatBuffers, so it should only be used when schema-less flexibility is required.

  5. What is FlatBuffers and when should I use it?

    master

    FlatBuffers is a memory-efficient serialization library designed for high-performance applications like games. Unlike traditional serialization formats that require a parsing/unpacking step to access data, FlatBuffers represents hierarchical data in a flat binary buffer that can be accessed directly.

    Key benefits for developers:

    • Zero-parsing access: You can access serialized data without unpacking it, meaning the only memory required is the buffer itself. In C++, this requires 0 additional allocations.
    • Memory Efficiency: It is highly suitable for mmap or streaming, as you only need parts of the buffer in memory at once.
    • Performance: Data access speed is close to raw struct access, involving only a single indirection (via a vtable) to support format evolution.
    • Schema Evolution: Supports optional fields, providing both forwards and backwards compatibility.
    • Strong Typing: Errors are caught at compile time rather than through error-prone runtime checks.
    • Low Overhead: It has a tiny code footprint and minimal dependencies.

  6. What is Flexbuffers?

    master

    Flexbuffers is a schema-less binary format developed by Google. Unlike standard FlatBuffers, it does not require a predefined schema. It is designed for high efficiency, allowing data to be accessed without parsing, copying, or memory allocation. This makes it ideal for memory-friendly use cases, such as mmap-ing large amounts of free-form data.

    Key features include:

    • Compact Encoding: Automatically sizes containers to their smallest possible representation (8, 16, 32, or 64 bits).
    • Zero-copy Access: Access data directly from the binary buffer.
    • Rust Integration: Supports Serde for automatic serialization of Rust data structures.

  8. How FlexBuffers Vectors are encoded

    master

    A vector is governed by a single bit width supplied by its parent.

    Untyped Vectors (SL_VECTOR)

    An untyped vector consists of:

    1. A size field (placed before the vector; an offset to the vector points to the first element, making the size field effectively at index -1).
    2. The elements of the vector.
    3. Type bytes (one uint8_t per element) following the elements. These type bytes always follow the vector, even if the elements are larger scalars.

    Example: A vector of uint8_t values 1, 2, 3 is encoded as:

    uint8_t 3, 1, 2, 3, 4, 4, 4

    (Where 3 is size, 1, 2, 3 are elements, and 4, 4, 4 are the type bytes).

    Typed Vectors

    Typed vectors omit the trailing type bytes because the type is determined by the parent. This is used for:

    • Inline signed/unsigned integers (TYPE_VECTOR_INT / TYPE_VECTOR_UINT)
    • Floats (TYPE_VECTOR_FLOAT)
    • Keys (TYPE_VECTOR_KEY)

    Additionally, fixed-length vectors of sizes 2, 3, or 4 (e.g., TYPE_VECTOR_INT2) exist for space savings when storing common data like colors.

  9. Serialize Unions in FlatBuffers

    master

    Unions in FlatBuffers allow a field to hold one of several different table types. When serializing a union, you must perform two distinct steps:

    1. Set the Type Field: Explicitly set the auto-generated _type field (e.g., EquippedType) using the generated enum that identifies which type is currently in the union.
    2. Set the Value Field: Provide the offset of the already-serialized object that belongs to the union.

    This ensures the reader knows which specific type to cast the data to when accessing the union field.

    // C# example: Setting a union field
Monser.AddEquippedType(builder, Equipment.Weapon);
Monster.AddEquipped(builder, axe.Value);
    // C++ example: Setting a union field
flatbuffers::Offset<Monster> orc = CreateMonster(
    builder, 
    &position, 
    mana, 
    hp, 
    name, 
    inventory, 
    Color_Red, 
    weapons, 
    Equipment_Weapon, // The type enum
    axe.Union(),      // The value offset
    path
);

  10. Understand FlatBuffer format components and interoperability

    master

    A FlatBuffer is a binary, in-memory format consisting of scalars aligned to their own size. To ensure cross-platform interoperability, FlatBuffers assumes:

    • Floating-point: Uses IEEE-754 format.
    • Signed integers: Uses two's complemented representation.
    • Endianness: The endianness for floating-point numbers is the same as for integers. All scalars are represented in little-endian format. While big-endian machines can use FlatBuffers, they will be slower due to byte-swap intrinsics.

    The format is defined by offsets and adjacency rather than fixed memory locations. This allows for optimization and means two different implementations might produce different binary layouts for the same input data.

  11. Access untrusted buffers safely

    master

    When reading FlatBuffers from untrusted sources (like a network), use the safe API to prevent arbitrary memory access. The safe functions verify the buffer's integrity before interpreting it.

    Safe functions (with verification):

    • root
    • size_prefixed_root
    • root_with_opts
    • size_prefixed_root_with_opts

    Unsafe functions (skip verification):

    • root_unchecked
    • size_prefixed_root_unchecked
    • root_with_opts_unchecked
    • size_prefixed_root_with_opts_unchecked

    Use the _unchecked versions only when processing large amounts of data from a trusted source (e.g., your own files on disk) to avoid the performance cost of verification.

  12. Use Reflection to read/write unknown FlatBuffer formats

    master

    FlatBuffers provides experimental support for reflection, allowing you to traverse and manipulate data without knowing the exact schema at compile time. This is achieved using a meta-schema (found in reflection/reflection.fbs) that describes schemas themselves.

    Workflow:

    1. Use flatc to write out parsed schemas as binary FlatBuffers using the meta-schema.
    2. Load these binary schemas at runtime to query fields and read/write data.
    3. Include <flatbuffers/reflection.h> to access the generated meta-schema code and helper functions.

    Example usage can be found in test.cpp/ReflectionTest().