Protocompile

repository·main·Indexed 43 days ago

https://github.com/bufbuild/protocompile

A pure Go parsing and linking engine for Protocol Buffers that serves as a high-performance, programmatic alternative to protoc. It powers the Buf CLI and ecosystem, working natively with the google.golang.org/protobuf runtime API to compile Protobuf source files into descriptors through a pipeline of parsing, linking, and option interpretation.

Tokens
3.8K
Snippets
11
Records
19
Agent score
55%

What's inside Protocompile

  1. What is Protocompile

    main
    Protocompile is a parsing and linking engine for Protocol Buffers written in pure Go. It serves as a high-performance alternative to the official protoc compiler and is the engine that powers the Buf CLI and ecosystem. It is designed to work natively with the modern Go Protobuf runtime API (google.golang.org/protobuf).
  2. How to use the Protocompile API

    main

    The primary entry point for most users is the root package github.com/bufbuild/protocompile. The main tool for compiling Protobuf source files into descriptors is the Compiler type.

    For more granular control, the library is organized into several sub-packages representing different stages of the compilation pipeline:

    • protocompile: Entry point for configuring and initiating compilation.
    • parser: Parses source code into an Abstract Syntax Tree (AST).
    • ast: Models the Protobuf AST.
    • linker: Links descriptor protos into more useful data structures and performs validation (e.g., checking type references and name uniqueness).
    • options: Interprets Protobuf options using linked data structures.
    • sourceinfo: Generates metadata mapping descriptor elements back to their original source locations and comments.
    • reporter: Provides error types and interfaces for reporting errors and warnings.
    • walk: Provides utilities for traversing descriptor hierarchies.
    • protoutil: General utility functions for interacting with descriptors.
  3. Migrating from `protoparse` to `protocompile`

    main

    If you are migrating from github.com/jhump/protoreflect/desc/protoparse, note the following changes:

    Standard Imports

    Unlike protoc, standard well-known files are not included by default. You must explicitly include them by wrapping your resolver with protocompile.WithStandardImports.

    Loading Files from a Map

    Instead of protoparse.FileContentsFromMap, use a protocompile.SourceResolver combined with protocompile.SourceAccessorFromMap as the accessor function.

    Parsing to AST

    Instead of Parser.ParseToAST, use parser.Parse from the parser sub-package. This returns an AST for the provided file contents.

    Parsing without Linking

    To replicate Parser.ParseFilesButDoNotLink, follow these steps:

    1. Parse the source using parser.Parse.
    2. Use parser.ResultFromAST to create a result containing a file descriptor proto.
    3. Interpret available options using options.InterpretUnlinkedOptions (note: some options, like custom ones, may remain uninterpreted).
    4. Generate source code info by calling sourceinfo.GenerateSourceInfo using the index returned from the previous step, and store the result in the file descriptor proto.
  4. What is a Resolver in protocompile

    main

    A Resolver is the mechanism the compiler uses to locate and load compilation inputs and their dependencies. It acts as the bridge between the compiler and your file system, network, or memory.

    A Resolver can return several types of data for a given query, which determines how much work the compiler must do:

    • Source code: The compiler will perform full parsing and compilation.
    • AST: The compiler skips the parsing step and proceeds with the rest of the compilation.
    • Descriptor proto: The compiler skips parsing and starts at the linking phase.
    • Descriptor: The compiler uses the descriptor as-is, performing no further work.
  5. Understand SearchResult priority and fields

    main

    A SearchResult contains information about a proto file. Only one field should be set. If multiple fields are set, the compiler follows a specific priority order (from highest to lowest preference):

    1. Desc (protoreflect.FileDescriptor): A fully linked descriptor. If present, the compiler does minimal work.
    2. ParseResult (parser.Result): Packages both an AST and a descriptor. This is more efficient than an AST and provides better error messages than a raw descriptor.
    3. Proto (*descriptorpb.FileDescriptorProto): A descriptor proto. The compiler will link this with dependencies.
    4. AST (*ast.FileNode): The abstract syntax tree. The compiler will convert this to a descriptor proto.
    5. Source (io.Reader): Raw source code. The compiler will parse this into an AST.
  6. How protocompile works

    main

    Protocompile is a high-performance native Go protobuf compiler that parses, validates, and generates fully-linked descriptors. Unlike the protoc CLI, it does not perform code generation.

    The compilation process follows these distinct phases:

    1. Parse into AST: Converting source code into an Abstract Syntax Tree (using parser.Parse).
    2. Convert AST to unlinked descriptor protos: Transforming the AST into unlinked descriptor messages (using parser.ResultFromAST).
    3. Link descriptor protos: Linking unlinked descriptors into "rich" descriptors (using linker.Link).
    4. Interpret custom options: Processing custom protobuf options (using options.InterpretOptions).
    5. Generate source code info: Producing source code information (using sourceinfo.GenerateSourceInfo).

    The package automatically selects the appropriate phases based on the input type provided by the Resolver. For example, providing source code triggers all phases, while providing a descriptor proto skips parsing and starts at the linking phase.

  7. Initialize a minimal Compiler

    main

    To create a basic compiler that resolves files from the local file system based on the current working directory, use the protocompile.Compiler struct with a protocompile.SourceResolver.

    By default, this minimal configuration:

    • Uses parallelism equal to the number of detected CPU cores.
    • Does not generate source code info in the resulting descriptors.
    • Fails fast at the first error encountered.
    compiler := protocompile.Compiler{
        Resolver: &protocompile.SourceResolver{},
    }
  8. Use the Compiler to compile protobuf files

    main

    The Compiler is the primary entrypoint for turning protobuf source files (or intermediate representations like ASTs or descriptor protos) into fully linked descriptors.

    To use it, you must provide a Resolver which handles locating source code and dependencies. The compilation process follows five steps: parsing into an AST, converting AST to descriptor protos, linking descriptors, interpreting options, and computing source code information.

    // Example usage (requires a Resolver implementation)
    compiler := &protocompile.Compiler{
        Resolver: myResolver,
    }
    
    // Compile specific files
    files, err := compiler.Compile(ctx, "path/to/file.proto", "path/to/another.proto")
    if err != nil {
        // handle error
    }
    // Use the resulting linker.Files
  9. Include standard proto imports with WithStandardImports

    main

    The WithStandardImports function wraps an existing Resolver and adds support for standard Google protobuf imports (like descriptor.proto).

    Note on Extension Declarations: As of v0.14.0, the standard imports returned by this function do not include extension declarations (e.g., in descriptor.proto) because they are only available in source and not in the embedded descriptors of the Protobuf Go module. If you need these declarations to prevent illegal re-definitions of custom features, use wellknownimports.WithStandardImports instead.

    // Wrap your existing resolver with standard imports
    resolver := protocompile.WithStandardImports(myBaseResolver)
  10. Use SourceResolver to load files from the file system or custom accessors

    main

    The SourceResolver resolves file names by returning their raw source code.

    • ImportPaths: If provided, all file paths are searched relative to these paths. If empty, paths are relative to the current working directory.
    • Accessor: An optional function to retrieve file contents. If nil, it defaults to os.Open.
    resolver := &protocompile.SourceResolver{
        ImportPaths: []string{"./proto", "/usr/local/include"},
        // Accessor is optional; defaults to os.Open
    }
  11. Configure the Compiler

    main

    The Compiler struct can be configured with the following fields:

    • Resolver (Required): Resolves path/file names into source code or intermediate representations. This is how the compiler loads files and their dependencies.
    • MaxParallelism: The maximum parallelism to use. If $\le 0$, it defaults to min(runtime.NumCPU(), runtime.GOMAXPROCS(-1)).
    • Reporter: A custom reporter.Reporter. If unspecified, a default reporter is used which fails compilation on the first error and ignores warnings.
    • SourceInfoMode: Controls how source code information (comments and positions) is generated.
    • RetainASTs: If true, ASTs are kept in the compilation results. If false, they are removed after use to save memory.
    • Symbols: A pointer to linker.Symbols. Providing this allows you to include symbols already known from previous compilations, enabling collision detection across multiple compilation batches.
  12. Handle Panics in the Compiler

    main

    If a Resolver implementation panics during compilation, the Compiler will recover and return a PanicError.

    To access the details of the panic (such as the file being processed and the stack trace), you should use errors.As to check for a *protocompile.PanicError.

    var panicErr *protocompile.PanicError
    if errors.As(err, &panicErr) {
        fmt.Printf("Panic in file %s: %v\n", panicErr.File, panicErr.Value)
        fmt.Printf("Stack trace: %s\n", panicErr.Stack)
    }