What is Protocompile
mainprotoc compiler and is the engine that powers the Buf CLI and ecosystem. It is designed to work natively with the modern Go Protobuf runtime API (google.golang.org/protobuf).repository·main·Indexed 43 days ago
https://github.com/bufbuild/protocompileA pure Go parsing and linking engine for Protocol Buffers that serves as a high-performance, programmatic alternative to protoc. It powers the Buf CLI and ecosystem, working natively with the google.golang.org/protobuf runtime API to compile Protobuf source files into descriptors through a pipeline of parsing, linking, and option interpretation.
protoc compiler and is the engine that powers the Buf CLI and ecosystem. It is designed to work natively with the modern Go Protobuf runtime API (google.golang.org/protobuf).The primary entry point for most users is the root package github.com/bufbuild/protocompile. The main tool for compiling Protobuf source files into descriptors is the Compiler type.
For more granular control, the library is organized into several sub-packages representing different stages of the compilation pipeline:
protocompile: Entry point for configuring and initiating compilation.parser: Parses source code into an Abstract Syntax Tree (AST).ast: Models the Protobuf AST.linker: Links descriptor protos into more useful data structures and performs validation (e.g., checking type references and name uniqueness).options: Interprets Protobuf options using linked data structures.sourceinfo: Generates metadata mapping descriptor elements back to their original source locations and comments.reporter: Provides error types and interfaces for reporting errors and warnings.walk: Provides utilities for traversing descriptor hierarchies.protoutil: General utility functions for interacting with descriptors.If you are migrating from github.com/jhump/protoreflect/desc/protoparse, note the following changes:
Unlike protoc, standard well-known files are not included by default. You must explicitly include them by wrapping your resolver with protocompile.WithStandardImports.
Instead of protoparse.FileContentsFromMap, use a protocompile.SourceResolver combined with protocompile.SourceAccessorFromMap as the accessor function.
Instead of Parser.ParseToAST, use parser.Parse from the parser sub-package. This returns an AST for the provided file contents.
To replicate Parser.ParseFilesButDoNotLink, follow these steps:
parser.Parse.parser.ResultFromAST to create a result containing a file descriptor proto.options.InterpretUnlinkedOptions (note: some options, like custom ones, may remain uninterpreted).sourceinfo.GenerateSourceInfo using the index returned from the previous step, and store the result in the file descriptor proto.A Resolver is the mechanism the compiler uses to locate and load compilation inputs and their dependencies. It acts as the bridge between the compiler and your file system, network, or memory.
A Resolver can return several types of data for a given query, which determines how much work the compiler must do:
A SearchResult contains information about a proto file. Only one field should be set. If multiple fields are set, the compiler follows a specific priority order (from highest to lowest preference):
Desc (protoreflect.FileDescriptor): A fully linked descriptor. If present, the compiler does minimal work.ParseResult (parser.Result): Packages both an AST and a descriptor. This is more efficient than an AST and provides better error messages than a raw descriptor.Proto (*descriptorpb.FileDescriptorProto): A descriptor proto. The compiler will link this with dependencies.AST (*ast.FileNode): The abstract syntax tree. The compiler will convert this to a descriptor proto.Source (io.Reader): Raw source code. The compiler will parse this into an AST.Protocompile is a high-performance native Go protobuf compiler that parses, validates, and generates fully-linked descriptors. Unlike the protoc CLI, it does not perform code generation.
The compilation process follows these distinct phases:
parser.Parse).parser.ResultFromAST).linker.Link).options.InterpretOptions).sourceinfo.GenerateSourceInfo).The package automatically selects the appropriate phases based on the input type provided by the Resolver. For example, providing source code triggers all phases, while providing a descriptor proto skips parsing and starts at the linking phase.
To create a basic compiler that resolves files from the local file system based on the current working directory, use the protocompile.Compiler struct with a protocompile.SourceResolver.
By default, this minimal configuration:
compiler := protocompile.Compiler{
Resolver: &protocompile.SourceResolver{},
}The Compiler is the primary entrypoint for turning protobuf source files (or intermediate representations like ASTs or descriptor protos) into fully linked descriptors.
To use it, you must provide a Resolver which handles locating source code and dependencies. The compilation process follows five steps: parsing into an AST, converting AST to descriptor protos, linking descriptors, interpreting options, and computing source code information.
// Example usage (requires a Resolver implementation)
compiler := &protocompile.Compiler{
Resolver: myResolver,
}
// Compile specific files
files, err := compiler.Compile(ctx, "path/to/file.proto", "path/to/another.proto")
if err != nil {
// handle error
}
// Use the resulting linker.FilesThe WithStandardImports function wraps an existing Resolver and adds support for standard Google protobuf imports (like descriptor.proto).
Note on Extension Declarations: As of v0.14.0, the standard imports returned by this function do not include extension declarations (e.g., in descriptor.proto) because they are only available in source and not in the embedded descriptors of the Protobuf Go module. If you need these declarations to prevent illegal re-definitions of custom features, use wellknownimports.WithStandardImports instead.
// Wrap your existing resolver with standard imports
resolver := protocompile.WithStandardImports(myBaseResolver)The SourceResolver resolves file names by returning their raw source code.
ImportPaths: If provided, all file paths are searched relative to these paths. If empty, paths are relative to the current working directory.Accessor: An optional function to retrieve file contents. If nil, it defaults to os.Open.resolver := &protocompile.SourceResolver{
ImportPaths: []string{"./proto", "/usr/local/include"},
// Accessor is optional; defaults to os.Open
}The Compiler struct can be configured with the following fields:
Resolver (Required): Resolves path/file names into source code or intermediate representations. This is how the compiler loads files and their dependencies.MaxParallelism: The maximum parallelism to use. If $\le 0$, it defaults to min(runtime.NumCPU(), runtime.GOMAXPROCS(-1)).Reporter: A custom reporter.Reporter. If unspecified, a default reporter is used which fails compilation on the first error and ignores warnings.SourceInfoMode: Controls how source code information (comments and positions) is generated.RetainASTs: If true, ASTs are kept in the compilation results. If false, they are removed after use to save memory.Symbols: A pointer to linker.Symbols. Providing this allows you to include symbols already known from previous compilations, enabling collision detection across multiple compilation batches.If a Resolver implementation panics during compilation, the Compiler will recover and return a PanicError.
To access the details of the panic (such as the file being processed and the stack trace), you should use errors.As to check for a *protocompile.PanicError.
var panicErr *protocompile.PanicError
if errors.As(err, &panicErr) {
fmt.Printf("Panic in file %s: %v\n", panicErr.File, panicErr.Value)
fmt.Printf("Stack trace: %s\n", panicErr.Stack)
}