vtprotobuf Documentation

vtprotobuf is an auxiliary plugin that must be run alongside the upstream protoc-gen-go generator. It does not replace it; instead, it generates fully-compatible auxiliary code (files ending in _vtproto.pb.go) to speed up serialization and deserialization without using reflection.

Requirements:

• Your project must use the ProtoBuf v2 API (google.golang.org/protobuf). It is not compatible with APIv1 generated code.
• You must pass the desired features via the --go-vtproto_opt flag during compilation.

# Example protoc invocation
protoc \
    --go_out=. --plugin protoc-gen-go="${GOBIN}/protoc-gen-go" \
    --go-grpc_out=. --plugin protoc-gen-go-grpc="${GOBIN}/protoc-gen-go-grpc" \
    --go-vtproto_out=. --plugin protoc-gen-go-vtproto="${GOBIN}/protoc-gen-go-vtproto" \
    --go-vtproto_opt=features=marshal+unmarshal+size \
    proto/your_file.proto

Twirp does not support custom marshalling/unmarshalling codecs by default. To use vtprotobuf optimizations with Twirp, you must perform a search-and-replace on the generated .twirp.go files after running protoc.

Specifically, replace calls to proto.Marshal(respContent) with respContent.MarshalVT() and replace proto.Unmarshal(buf, reqContent) with reqContent.UnmarshalVT(buf).

for twirp in $${dir}/*.twirp.go; \ 
do \ 
  echo 'Updating' $${twirp}; \ 
  sed -i '' -e 's/respBytes, err := proto.Marshal(respContent)/respBytes, err := respContent.MarshalVT()/g' $${twirp}; \ 
  sed -i '' -e 's/if err = proto.Unmarshal(buf, reqContent); err != nil {/if err = reqContent.UnmarshalVT(buf); err != nil {/g' $${twirp}; \ 
done; \ 

To enable vtprotobuf encoding in DRPC, pass the github.com/planetscale/vtprotobuf/codec/drpc package to the protolib flag during your protoc-gen-go-drpc invocation.

protoc --go_out=. --go-vtproto_out=. --go-drpc_out=. --go-drpc_opt=protolib=github.com/planetscale/vtprotobuf/codec/drpc

The protoc-gen-go-vtproto compiler generates helper methods rather than overwriting default marshalling code. To opt-in to optimized (de)serialization in gRPC, you must register the codec provided by github.com/planetscale/vtprotobuf/codec/grpc.

It is recommended to perform a blank import of the default google.golang.org/grpc/encoding/proto package to ensure the vtprotobuf codec replaces it. The provided codec will attempt to use optimized codegen for all ProtoBuf messages it encounters.

package servenv

import (
	"github.com/planetscale/vtprotobuf/codec/grpc"
	"google.golang.org/grpc/encoding"
	_ "google.golang.org/grpc/encoding/proto"
)

func init() {
	encoding.RegisterCodec(grpc.Codec{})
}

If you enable the pool feature, you must specify which messages should be pooled. You can do this in two ways:

1. Via .proto file options

Use the (vtproto.mempool) = true option on the message definition.

import "github.com/planetscale/vtprotobuf/vtproto/ext.proto";

message SampleMessage {
    option (vtproto.mempool) = true;
    string name = 1;
}

2. Via CLI flags

Pass the --go-vtproto_opt=pool=<import>.<message> flag for each message to be pooled.

--go-vtproto_opt=features=marshal+unmarshal+size+pool \
--go-vtproto_opt=pool=vitess.io/vitess/go/vt/proto/query.Row

Install the protoc-gen-go-vtproto plugin using go install. This plugin is used as an auxiliary generator alongside protoc-gen-go to provide optimized (de)serialization code.

go install github.com/planetscale/vtprotobuf/cmd/protoc-gen-go-vtproto@latest

To prevent vtprotobuf generated methods from being compiled into your binary by default (e.g., if you want to allow users to opt-in), use the --vtproto_opt=buildTag=<tag> flag.

When this is used, the generated code will only be included if the specified build tag is provided during go build. It is recommended to use type assertions in your code to safely call these methods so your project remains compilable even without the tag.

To use vtprotobuf with Connect, you must implement a custom codec that handles messages based on their type (since Connect internally uses types like Status that lack vtprotobuf helpers). Once implemented, pass your codec to both the handler and client constructors using the connect.WithCodec option.

package main

import (
	"net/http"

	"github.com/bufbuild/connect-go"
	"github.com/foo/bar/pingv1connect"
	"github.com/myorg/myproject/codec/mygrpc"
)

func main() {
	mux := http.NewServeMux()
	mux.Handle(pingv1connect.NewPingServiceHandler(
		&PingServer{},
		connect.WithCodec(mygrpc.Codec{}), // Add connect option to handler.
	))
	// handler serving ...

	client := pingv1connect.NewPingServiceClient(
		http.DefaultClient,
		"http://localhost:8080",
		connect.WithCodec(mygrpc.Codec{}), // Add connect option to client.
	)
	/// client code here ...
}

You can automate vtprotobuf generation by adding go-vtproto as a plugin in your buf.gen.yaml configuration file. After installing the protoc-gen-go-vtproto binary, running buf generate will automatically include the optimized helpers in your generated code.

version: v1
managed:
  enabled: true
  # ...
plugins:
  - plugin: buf.build/protocolbuffers/go
    out: ./
    opt: paths=source_relative
  - plugin: go-vtproto
    out: ./
    opt: paths=source_relative

The unique field option can be applied to string fields to intern them using unique.Make (requires Go 1.23+). This is useful for reducing memory usage for repetitive strings. Note that unmarshal_unsafe takes precedence over unique if both are enabled.

import "github.com/planetscale/vtprotobuf/vtproto/ext.proto";

message Label {
    string name  = 1 [(vtproto.options).unique = true];
    string value = 2 [(vtproto.options).unique = true];
}