DOCSEARCHbyAlgolia
Submit Docs

Buf Toolchain for Protobuf

repository·main·Indexed 11 days ago

https://github.com/bufbuild/buf

A modern toolchain for Protobuf that replaces protoc with a faster, module-aware workflow. Buf provides capabilities for linting, formatting, breaking-change detection, and managed code generation. It includes a CLI for managing modules and workspaces, and integrates with the Buf Schema Registry (BSR) for hosting schemas, remote plugins, and SDK distribution.

Tokens
29K
Snippets
134
Records
148
Agent score
92%

What's inside Buf

  1. What is a Protobuf source path?

    main

    A Protobuf source path is a SourceCodeInfo.Location path, represented as a variable-length array of integers. It identifies a specific Protobuf definition by forming a path from the FileDescriptorProto to the target definition.

    Each element in the array represents either a field number from a descriptor proto or an index for repeated types (such as messages, enums, services, or extensions) on the FileDescriptorProto.

    Example breakdown of [4, 0, 2, 0, 1]:

    • 4: Field number of message_type on FileDescriptorProto (a repeated field).
    • 0: Index of the specific message.
    • 2: Field number of field on DescriptorProto (a repeated field).
    • 0: Index of the specific field.
    • 1: Field number of name on FieldDescriptorProto.

  2. How Buf modules and workspaces work

    main

    Buf organizes Protobuf files using two main abstractions:

    1. Module: A directory tree of .proto files defined by a buf.yaml file. The buf.yaml file configures how the module is built, linted, and checked for breaking changes.
    2. Workspace: A collection of one or more modules.

    A typical buf.yaml (version v2) defines the module paths and the rules to apply:

    version: v2
modules:
  - path: proto
lint:
  use:
    - STANDARD
breaking:
  use:
    - FILE

  3. How associated source paths work

    main

    The protosourcepath package identifies "associated" paths for a given source path. These are used to find related metadata, such as comments or lint rule ignores, at different levels of a Protobuf declaration.

    Parent paths

    Parent paths are valid source paths that represent "complete" Protobuf declarations that are equal to or closer to the FileDescriptorProto than the input path. A complete declaration starts at a keyword (e.g., message) or label/name and terminates at an opening brace or semicolon.

    • Top-level declarations: The complete declaration itself is a parent path.
    • Nested declarations: The complete declaration of the nested item and the complete declarations of all its parent types are considered parent paths.
    • Specific attributes: For attributes like name or label, the complete declaration of the definition they belong to is a parent path.

    Child paths

    Child paths are valid source paths that are not complete Protobuf declarations and are not closer to the FileDescriptorProto than the input path.

    • If a path is not a complete declaration, it is considered its own child path.
    • Other attributes of the same definition (e.g., field number, label, type) are child paths.
    • Associated child paths of the parent type (e.g., the message name) are also included.

  4. Install the Buf CLI

    main

    You can install the buf CLI using Homebrew, which also installs the protoc-gen-buf-breaking and protoc-gen-buf-lint binaries along with shell completion scripts.

    Other supported installation methods include npm, Windows, Docker, binary downloads, tarballs, source builds, and Minisign verification. Refer to the official installation documentation for full details.

    brew install bufbuild/buf/buf

  5. Generate code with `buf generate`

    main

    Instead of using manual protoc shell commands, buf generate uses a versioned configuration file (typically buf.gen.yaml). This allows you to define plugins (local or remote), output directories, and options in a declarative way.

    Remote Plugins: You can use plugins hosted on the Buf Schema Registry (BSR), which removes the need to install generator binaries on every developer machine or CI runner.

    Managed Mode: This allows you to keep language-specific file options (like go_package) out of your .proto files while ensuring consumers get the correct generated package names.

    version: v2
clean: true
managed:
  enabled: true
  override:
    - file_option: go_package_prefix
      value: github.com/acme/weather/gen/go
plugins:
  - remote: buf.build/protocolbuffers/go
    out: gen/go
    opt: paths=source_relative
  - remote: buf.build/connectrpc/gosimple
    out: gen/go
    opt:
      - paths=source_relative
      - simple
inputs:
  - directory: proto

  6. Publish modules to the Buf Schema Registry (BSR)

    main

    The Buf Schema Registry (BSR) is a Protobuf-aware registry that stores modules, verifies compilation, renders documentation, and hosts remote plugins.

    You can publish your modules to the BSR using the buf push command. Once published, consumers can:

    • Depend on the schema as a BSR module.
    • Install generated SDKs via standard package managers (e.g., go get, npm install, pip install, etc.).
    • Inspect services, messages, and fields via hosted BSR documentation.
    buf push

  7. Initialize and run Buf checks in a new project

    main

    To set up a new Protobuf repository with Buf, initialize a configuration and run the standard suite of checks (build, format, lint, and breaking-change detection).

    Note: The buf breaking command below uses a Git branch comparison as an example.

    buf config init
buf build
buf format -w
buf lint
buf breaking --against '.git#branch=main'

  8. How `buf curl` shell completion works

    main

    The buf curl command provides intelligent shell completion for the URL positional argument. It attempts to complete service and method names by searching through sources in the following order of precedence:

    1. Explicit --schema flag: If you provide schemas via --schema, completion is built from those definitions.
    2. Live Server Reflection: If the server is reachable and supports reflection, buf curl uses gRPC reflection to provide real-time completions.
    3. Local Buf Module: If no schema is provided and reflection fails, it walks up from the current working directory to find a buf.yaml or buf.work.yaml to use as a local source.

    Completion Behavior:

    • Package Segments: A single tab press will skip unambiguous package segments (e.g., if acme.foo. is the only match, it completes to the next segment).
    • Service Names: Completes to the full service name followed by a / (e.g., acme.foo.v1.FooService/).
    • Method Names: Once a service is selected, it completes the available method names for that service.

  9. How `buf registry login` authentication flows work

    main

    The buf registry login command supports three distinct authentication methods depending on the environment:

    1. Browser Flow (Default): Uses the OAuth2 device authorization grant. It opens your default web browser to a verification URI. Once you authorize the device in the browser, the CLI automatically retrieves the token.
    2. Interactive Prompt (--prompt): Used when a browser cannot be opened (e.g., remote SSH sessions). It prompts you to enter a BSR token manually. This requires a TTY.
    3. Stdin Flow (--token-stdin): Used for non-interactive environments like CI/CD pipelines. It reads the token directly from standard input.

    Note: The --token-stdin and --prompt flags are mutually exclusive.

  10. Use flag files with `@` in `buf protoc`

    main

    To avoid long command lines, you can use flag files. By prefixing an argument with @, buf protoc will read the contents of that file and treat each line as a new flag or argument. This allows for recursive flag loading and complex configurations.

    # If flags.txt contains:
# -I=./protos
# --go_out=./gen

# Use it in the command line:
buf protoc @flags.txt service.proto

  11. Print the dependency graph with `buf dep graph`

    main

    The buf dep graph command prints the dependency graph of a specified source or module. This is useful for visualizing how your Protobuf modules relate to one another, including dependencies from the Buf Schema Registry (BSR).

    To visualize the graph as an image, you can pipe the output to the dot tool from Graphviz.

    Prerequisites To generate image files, you must have graphviz installed. On macOS, you can install it via Homebrew:

    brew install graphviz

    Usage Example To generate a PNG image of your dependency graph:

    buf dep graph | dot -Tpng >| graph.png && open graph.png

  12. Use `buf curl` to invoke RPC methods

    main

    The buf curl command provides a curl-like interface for interacting with gRPC/Connect servers. It can be used to list services, list methods, or invoke specific RPC methods by providing a URL that includes the service and method path.

    To invoke an RPC, you provide a URL in the format https://<host>/<service>/<method>. The command handles schema resolution (via --schema or local modules), server reflection, and data encoding/decoding.

    Key capabilities include:

    • Service/Method Discovery: Use --list-services or --list-methods to explore available endpoints.
    • Schema Resolution: Use --schema to provide Protobuf definitions if the server does not support reflection.
    • Protocol Support: Supports HTTP/2, HTTP/3, and gRPC via reflection.
    • Authentication: Supports custom headers and basic credentials.
    # Example: Invoking an RPC method
buf curl https://api.example.com/acme.foo.v1.FooService/GetFoo

# Example: Listing services on a server
buf curl --list-services https://api.example.com

# Example: Using a local schema for discovery/invocation
buf curl --schema . https://api.example.com/acme.foo.v1.FooService/GetFoo