exprotobuf

repository·master·Indexed 19 days ago

https://github.com/bitwalker/exprotobuf

A library for working with Google Protocol Buffers natively in Elixir. It generates Elixir module and struct definitions from .proto schemas, providing tools for encoding and decoding protobuf data. Features include support for nested messages, enums, oneof fields, and flexible module generation via injection or namespacing.

Tokens
5.5K
Snippets
22
Records
25
Agent score
67%

What's inside exprotobuf

  1. Migrate from automatic imports to explicit file lists in `use Protobuf`

    master

    In older versions of exprotobuf, import statements within .proto files were automatically resolved and their definitions were copied into the namespace of the primary message. This behavior is deprecated.

    In current versions, you must explicitly provide a list of all relevant .proto files to the from: option in use Protobuf. This ensures that shared definitions (like enums or messages used across multiple files) are parsed once and result in a single, shared Elixir module rather than duplicate modules in different namespaces.

    ## The New Behavior
    
    ```elixir
    defmodule Test do
      use Protobuf, from: ["./test/basic.proto", "./test/colors.proto"]
    end
  2. Install exprotobuf

    master

    To use exprotobuf in your Elixir project, add it to your mix.exs dependencies and include it in your application's application list.

    1. Add to deps:
    defp deps do
      [{:exprotobuf, "~> x.x.x"}]
    end
    1. Run mix deps.get.
    2. Add to applications list:
    def application do
      [applications: [:exprotobuf]]
    end
    defp deps do
      [{:exprotobuf, "~> x.x.x"}]
    end
    
    def application do
      [applications: [:exprotobuf]]
    end
  3. Customize generated module names and namespaces

    master

    By default, exprotobuf respects the namespace of the messages. You can control how modules are nested using the following options:

    • use_package_names: true: Uses the package name defined in the .proto file as a module prefix.
    • namespace: :<atom>: Forces all generated modules into a specific top-level namespace.

    Example using use_package_names: If example.proto has package world; and a message Example, calling use Protobuf, from: "...", use_package_names: true allows you to access it via Definitions.World.Example.

    Example using namespace: To flatten everything into a specific namespace (e.g., Elixir), use namespace: :Elixir alongside use_package_names: true.

    defmodule Definitions do
      use Protobuf, from: Path.wildcard("protobufs/*.proto"), use_package_names: true, namespace: :Elixir
    end
    
    # Access via:
    World.Example.new(continent: :EUROPE)
  4. How `def_message` handles module injection

    master

    The def_message/5 macro provides two distinct ways to structure your generated Protobuf code based on the inject option:

    1. Direct Injection (inject: true)

    When inject: true is passed, the macro defines the struct and all methods (like encode/1, decode/1, and new/1) directly within the module where the macro is called. The module becomes the message itself.

    2. Nested Module (inject: false)

    When inject: false is passed, the macro creates a new nested module. This is the preferred way to organize complex schemas. The macro uses a use_in mechanism to allow the generated nested module to be associated with a root namespace.

    This allows you to define a schema like this:

    defmodule MyApp.Protobuf do
      use Protobuf.DefineMessage
    
      def_message(:Settings, [...], inject: false)
    end
    
    # The message is now available at:
    # MyApp.Protobuf.Settings
  5. Inject specific types from a larger subset

    master

    When using Protobuf.Builder.define/2, you can use the only option in your Protobuf.Config to restrict the generation to a specific subset of types. This is useful when you have a large .proto file but only want to expose certain messages or enums to your Elixir application.

    If a type is a child of a type listed in only (determined by dot-notation in the name), it may also be included depending on the internal logic. The builder handles namespace fixing for these injected types to ensure they reside under the correct module path.

    config = %Protobuf.Config{only: [:MySpecificMessage, :SomeEnum], namespace: :MyNamespace}
    Protobuf.Builder.define(msgs, config)
  6. Extend generated modules via `use_in`

    master

    If you need to add custom behavior (functions/macros) to a generated Protobuf module, use use_in "<ModuleName>", <ModuleToUse>.

    Warning: Because the struct for the target module is not defined until the use Protobuf call completes, you cannot rely on the struct type in your functions for compile-time guarantees. It is recommended to use :inject instead whenever possible. If you use use_in, you must interact with the generated structs using the standard Maps API.

    defmodule Messages do
      use Protobuf, "message Msg { required uint32 v = 1; }"
    
      defmodule MsgHelpers do
        defmacro __using__(_opts) do
          quote do
            def convert_to_tuple(msg) do
              msg |> Map.to_list() |> Enum.map(&{&2, &1}) |> Enum.sort() |> Enum.map(&{&1, &2}) |> Enum.reduce([], fn {k, v}, acc -> [v | acc] end) |> Enum.reverse |> list_to_tuple
            end
          end
        end
      end
    
      use_in "Msg", MsgHelpers
    end
  7. Filter loaded types with the `only` option

    master

    When loading a large .proto file, you can restrict the generated modules to a specific subset using the only: [:TypeA, :TypeB] option.

    Constraints:

    • You can only combine :only with :inject if :only contains exactly one type (because a module can only hold one struct).
    • Selecting a child type that depends on a parent type not included in the :only list may cause compilation or runtime errors.
    defmodule Messages do
      use Protobuf, from: Path.expand("../proto/messages.proto", __DIR__), only: [:TypeA, :TypeB]
    end
  8. Inject a Protobuf definition into an existing module

    master

    If you want to avoid creating nested modules and instead want the Protobuf types to be defined directly within your current module, use the inject: true option.

    This is useful for single types or when you want a cleaner top-level module structure. Note that when inject: true is used, the message is no longer a nested module but is part of the module where use Protobuf is called.

    defmodule Msg do
      use Protobuf, from: Path.expand("../proto/messages.proto", __DIR__), inject: true
    
      def update(msg, key, value), do: Map.put(msg, key, value)
    end
    
    # Usage
    %Msg{v: :V1}
  9. Define Protobuf modules from files

    master

    You can load Protobuf definitions from a single file or a set of files using the from: option in the use Protobuf statement.

    From a single file

    defmodule Messages do
      use Protobuf, from: Path.expand("../proto/messages.proto", __DIR__)
    end

    From multiple files (wildcard)

    Loading multiple files allows definitions (like enums or messages) to be shared across different .proto files.

    defmodule Protobufs do
      use Protobuf, from: Path.wildcard(Path.expand("../definitions/**/*.proto", __DIR__))
    end
    defmodule Messages do
      use Protobuf, from: Path.expand("../proto/messages.proto", __DIR__)
    end
    
    defmodule Protobufs do
      use Protobuf, from: Path.wildcard(Path.expand("../definitions/**/*.proto", __DIR__))
    end
  10. Define Protobuf modules from a string

    master

    You can define Protobuf schemas directly within an Elixir module using the use Protobuf, "<schema_string>" syntax. This generates modules and structs for the types defined in the string.

    Key Behaviors:

    • Nested Messages: Generates nested modules/structs (e.g., Messages.Msg containing Messages.Msg.SubMsg).
    • Enums: Generates a module with atom(x) and value(x) functions rather than a struct.
    • oneof: Represented as tuples in the struct (e.g., {:field_name, value}).
    • Encoding/Decoding: Use .new/1 to create a struct, .encode/1 to get binary data, and .decode/1 to parse binary data back into a struct.
    defmodule Messages do
      use Protobuf, """
        message Msg {
          message SubMsg {
            required uint32 value = 1;
          }
    
          enum Version {
            V1 = 1;
            V2 = 2;
          }
    
          required Version version = 2;
          optional SubMsg sub = 1;
        }
      """
    end
    
    # Usage
    msg = Messages.Msg.new(version: :V2)
    encoded = Messages.Msg.encode(msg)
    original = Messages.Msg.decode(encoded)
  11. Configure the Protobuf parser behavior

    master

    The Protobuf.Config struct is used to define how the parser behaves when generating Elixir modules from Protocol Buffer definitions. You can use these options to control namespaces, specific type loading, and injection behavior.

    # Example of a configuration struct
    config = %Protobuf.Config{
      namespace: MyProject.Protobuf,
      schema: "path/to/schema.proto",
      only: ["MyMessage"],
      inject: true,
      use_google_types: true
    }
  12. Handle Protobuf.Parser.ParserError

    master

    When parsing fails, Protobuf.Parser raises a Protobuf.Parser.ParserError.

    One common error case is a Reference to undefined message or enum. The error message will explicitly identify the missing type and the location of the invalid reference. For example: Reference to undefined message or enum TypeName at Path.To.Field.

    Other errors may be returned as raw strings or as the result of Macro.to_string/1 on the underlying error structure.