DOCSEARCHbyAlgolia
Submit Docs

protozero

repository·master·Indexed 18 days ago

https://github.com/mapbox/protozero

A high-performance, minimalistic C++ library for encoding and decoding Protocol Buffers, optimized for zero-copy parsing and minimal memory allocation. Unlike standard Google Protobuf, protozero does not read .proto files, requiring developers to manually translate schemas into code. It provides low-level building blocks for stable schemas where performance and lazy decoding are critical.

Tokens
7.4K
Snippets
22
Records
36
Agent score
58%

What's inside protozero

  1. Overview of Protozero

    master

    Protozero is a minimalistic, high-performance C++ protocol buffer decoder and encoder. It is designed for scenarios requiring zero-copy parsing and minimal run-time memory allocation.

    Important Design Note: Protozero is a low-level building block. Unlike the standard Google Protobuf implementation, it does not read .proto files. Instead, developers must manually translate the .proto schema into code. This makes it ideal for stable schemas where performance and lazy decoding are critical, but unsuitable if your schema changes frequently or if you rely on generated C++ APIs from protoc.

  2. How Protozero works and its limitations

    master

    Protozero operates by having the developer manually implement the logic described in a .proto file. Because it does not parse .proto files directly, the following information is not available to the library:

    • Field names: You must handle field identification manually.
    • Enum names: You must use the underlying integer values.
    • Default values: These are not provided by the library.
    • Field types: The library does not know what types to expect; you must supply the correct types in your code. While the library performs some assert() checks, most type validation must be handled by the user.

    Safety: The library guarantees it will not overrun the provided buffer, but all other validation (type checks, value ranges, etc.) must be implemented in your application code.

  4. Understand Protozero limitations

    master

    Before using Protozero, be aware of the following constraints:

    • No Streaming Support: A protobuf message must fit entirely into memory to be parsed.
    • Size Limits: The length of a string, bytes, or submessage cannot exceed $2^{31}-1$.
    • Map Support: There is no native support for maps, but they can be used by following the standard Protobuf map encoding (treating them as repeated messages of key/value pairs).

  5. Use PROTOZERO_USE_VIEW to substitute data_view

    master

    Protozero provides a protozero::data_view class (compatible with std::string_view) to provide a more intuitive interface via .data() and .size() instead of using .first and .second on a std::pair.

    If you are using C++17 or have a similar class available, you can substitute the internal data_view by defining the PROTOZERO_USE_VIEW macro with the name of your preferred class before including types.hpp.

    #define PROTOZERO_USE_VIEW std::string_view
#include <protozero/types.hpp>

  6. Understand `pbf_reader` lifetime and memory

    master

    The protozero::pbf_reader is a lightweight value type (approx. 24 bytes) that can be copied or moved trivially.

    CRITICAL: pbf_reader stores a pointer into the input data provided during construction. You must ensure that the underlying data buffer (e.g., the std::string or char* passed to the constructor) remains valid for the entire lifetime of the pbf_reader object.

  7. Build and run tests

    master

    Protozero includes extensive tests. You can build and run them using CMake and ctest. Note that writer tests require the Google Protobuf library to be present on your system, while unit and reader tests do not.

    mkdir build
cd build
cmake ..
make

# Run the tests
ctest

  8. Handle repeated fields using `tag_and_type()`

    master

    Protozero does not automatically enforce the rule that non-repeated fields appearing multiple times should return the last value, nor does it automatically concatenate packed and unpacked repeated fields with the same tag. To handle these cases manually, use the tag_and_type() method on pbf_reader or pbf_message to distinguish between packed and unpacked encodings.

    pbf_message uses an enum for field tags, while pbf_reader uses numeric tags.

    enum class ExampleMsg : protozero::pbf_tag_type {
    repeated_uint32_x = 1
};

std::string data = ...;
protozero::pbf_message<ExampleMsg> message{data};
while (message.next()) {
    switch (message.tag_and_type()) {
        case tag_and_type(ExampleMsg::repeated_uint32_x, pbf_wire_type::length_delimited): {
                auto xit = message.get_packed_uint32();
                // handle packed field
            }
            break;
        case tag_and_type(ExampleMsg::repeated_uint32_x, pbf_wire_type::varint): {
                auto x = message.get_uint32();
                // handle unpacked field
            }
            break;
        default:
            message.skip();
    }
}

  9. Install Protozero

    master

    To install Protozero, use the CMake build system. After running the CMake configuration step, run make install to install the header files to /usr/local/include/protozero.

    If you are integrating Protozero into a project that also uses CMake, copy the cmake/FindProtozero.cmake file into your project and use it in your build configuration.

    # After the CMake step
make install

  10. Write nested sub-messages efficiently

    master

    There are two ways to handle sub-messages with pbf_writer:

    Method 1: Separate Buffers

    Create a separate pbf_writer for the sub-message using its own std::string buffer, then add that buffer to the parent message using pbf_parent.add_message(tag, buffer_sub).

    Method 2: In-place (Recommended)

    To avoid extra allocations, you can write the sub-message directly into the parent's buffer.

    1. Open a new scope.
    2. Initialize a new pbf_writer by passing the pbf_parent and the field tag: protozero::pbf_writer pbf_sub{pbf_parent, tag};.
    3. Add fields to pbf_sub within that scope.
    4. When the scope closes, the pbf_sub destructor automatically calculates the sub-message length and updates the parent's buffer.

    Rollback: If you need to abandon a sub-message, call pbf_sub.rollback() within the scope. Do not attempt to call add_* functions on the sub-message after a rollback.

    std::string data;
protozero::pbf_writer pbf_parent{data};

// Add fields to parent
pbf_parent.add_uint32(1, 100);

// Write sub-message in-place
{
    // Pass parent and the tag for the sub-message
    protozero::pbf_writer pbf_sub{pbf_parent, 2};

    pbf_sub.add_uint32(1, 500);
    // ... more sub-message fields

} // pbf_sub destructor writes the length here

// Continue with parent
pbf_parent.add_uint32(3, 300);

  11. Install and include Protozero

    master

    Protozero is a header-only library. To use it, you need a C++11-capable compiler. Copy the contents of the include/protozero directory into your project's include path. You can then include files using the following pattern:

    #include <protozero/FILENAME.hpp>
    #include <protozero/FILENAME.hpp>

  12. Reserve memory when writing messages

    master

    To optimize performance when writing messages, you can pre-allocate memory:

    1. On the underlying container: Call std::string::reserve() on the string you pass to a pbf_writer or pbf_builder.
    2. On the Protozero object: Call .reserve(n) directly on a pbf_writer or pbf_builder. Note that this reserves n bytes in addition to the current size of the message.