zig-protobuf

zig-protobuf generates service interfaces using a VTable (virtual table) pattern. Instead of a single fixed implementation, the generator produces a function that takes two comptime parameters: UserDataType (your server's context/state) and ErrorSet (the errors your service can return).

This function returns a struct type containing function pointers for each RPC method defined in your .proto file. This approach provides type safety for your custom state and errors while ensuring zero runtime overhead because polymorphism is resolved at compile time.

Note: This only generates the service interface. You must implement your own transport layer (e.g., gRPC over HTTP/2) and server logic.

// The generated function signature pattern
pub fn MyService(comptime UserDataType: type, comptime ErrorSet: type) type {
    return struct {
        pub const service_name = "MyService";

        // Example method signature
        UnaryCall: *const fn(userdata: *UserDataType, request: Request) ErrorSet!Response,
    };
}

While MyMessage.decode materializes a full message in memory, every generated message also provides a StreamDecoder. This is a zero-allocation pull parser that iterates through a std.Io.Reader one wire field at a time.

When to use it

• Large or deeply nested messages.
• Embedded systems with limited memory.
• Multiplexed IO where you want to avoid buffering the entire message.

How it works

Call next() to receive an Event.

• Scalars: Returned by value.
• Oneof: Leaf cases are flattened into their own variants.
• Length-delimited fields (string, bytes, submessages): Returned as a *std.Io.Reader bound to that field's bytes. You can recurse into these with another StreamDecoder or copy the bytes out.
• Repeated fields: Emits one event per element.

Critical Caveat

Do not copy the StreamDecoder after initialization. The *std.Io.Reader returned for length-delimited fields points back into the decoder instance itself.

var sd = MyMessage.StreamDecoder.init(&reader);
while (try sd.next()) |item| switch (item) {
    .some_scalar => |v| { ... },                  // value, by value
    .some_string => |limited| {                   // limited: *std.Io.Reader
        var buf: [64]u8 = undefined;
        const n = try limited.readSliceShort(&buf);
        ...
    },
    .some_submessage => |limited| {               // recurse without allocating
        var inner = SubMessage.StreamDecoder.init(limited);
        while (try inner.next()) |x| switch (x) { ... };
    },
    // repeated fields (packed or not) emit one event per element
    .some_repeated => |v| { ... },
};

To implement a service, follow these steps:

1. Define your context: Create a struct (UserDataType) to hold server state like allocators or database connections.
2. Define your errors: Create an error set (ErrorSet) specific to your service logic.
3. Instantiate the VTable type: Call the generated service function with your types.
4. Implement methods: Create functions that match the required signatures and assign them to a VTable instance.
5. Initialize the VTable: Create an instance of the struct with your implementations.

Example implementation flow:

const service = @import("example.pb.zig");
const std = @import("std");

const MyUserData = struct {
    allocator: std.mem.Allocator,
    connection_id: u64,
};

const MyErrors = error{
    InvalidRequest,
    ServiceUnavailable,
};

const MyServiceVTable = service.MyService(MyUserData, MyErrors);

const unaryCallImpl = struct {
    fn call(userdata: *MyUserData, request: service.Request) MyErrors!service.Response {
        return service.Response{ .result = "success" };
    }
}.call;

const myServiceVTable = MyServiceVTable{
    .UnaryCall = unaryCallImpl,
    // ... other methods
};

You can automate the generation of Zig code from .proto files by creating a custom build step in build.zig using protobuf.RunProtocStep. This step can be exposed as a command (e.g., zig build gen-proto).

Key configuration options for RunProtocStep.create:

• destination_directory: The LazyPath where generated files will be saved.
• protoc: (Optional) A LazyPath to a specific protoc binary. If omitted, the library will attempt to download the official Google release.
• source_files: An array of paths to your .proto files.
• include_directories: An array of directories to include for imports.
• preserve_unknown_fields: A boolean (defaults to false) that determines if unknown fields are preserved during binary encode/decode round trips.

const protobuf = @import("protobuf");

pub fn build(b: *std.Build) !void {
    const protobuf_dep = b.dependency("protobuf", .{
        .target = target,
        .optimize = optimize,
    });
    
    const gen_proto = b.step("gen-proto", "generates zig files from protocol buffer definitions");

    const protoc_step = protobuf.RunProtocStep.create(protobuf_dep.builder, target, .{
        .destination_directory = b.path("src/proto"),
        .source_files = &.{ 
            b.path("protocol/all.proto"),
        },
        .include_directories = &.{},
        .preserve_unknown_fields = false,
    });

    gen_proto.dependOn(&protoc_step.step);
}

Server Streaming

In a server streaming RPC, you receive a writer_queue. You loop through your data and call try writer_queue.write(response) for each item.

Client Streaming

In a client streaming RPC, you receive a reader_queue. You use a while (try reader_queue.read()) |request| loop to process all incoming messages before returning a single final response.

// Server Streaming Example
fn serverStreamImpl(
    userdata: *MyUserData,
    request: service.Request,
    writer_queue: *std.Io.Queue(service.Response)
) MyErrors!void {
    for (0..10) |i| {
        const response = service.Response{ .result = "item" };
        try writer_queue.write(response);
    }
}

// Client Streaming Example
fn clientStreamImpl(
    userdata: *MyUserData,
    reader_queue: *std.Io.Queue(service.Request)
) MyErrors!service.Response {
    var count: usize = 0;
    while (try reader_queue.read()) |request| {
        count += 1;
    }
    return service.Response{ .result = "done" };
}