betterproto

repository·master·Indexed 23 days ago

https://github.com/danielgtaylor/python-betterproto

A modern Protobuf and gRPC code generator and library for Python (3.7+) that produces idiomatic, type-safe, and async-ready code using dataclasses. It supports Protobuf 3, binary and JSON serialization, and generates async gRPC stubs compatible with grpclib. Key features include Mypy type checking support, Pydantic model generation, and mapping of Google well-known types to standard Python types like datetime and timedelta.

Tokens
16.1K
Snippets
37
Records
97
Agent score
78%

What's inside betterproto

  1. Overview of betterproto features

    master

    betterproto is a protobuf compiler and interpreter designed to improve the experience of using Protobuf and gRPC in Python. It generates readable, understandable, and idiomatic Python code using modern language features.

    Key features include:

    • Serialization: Generated messages support both binary and JSON serialization.
    • Pythonic Types: Messages utilize relevant Python types such as Enum, datetime, and timedelta objects.
    • Async Support: Provides async/await support for both gRPC Clients and Servers.
    • Code Quality: Generates modern, readable, and idiomatic Python code.
  2. Overview of betterproto features and supported environments

    master

    betterproto provides an improved experience for Protobuf 3 and gRPC in modern Python (3.7+) by generating idiomatic, readable, and type-hinted code.

    Key features include:

    • Protobuf 3 & gRPC support: Built-in binary and JSON serialization.
    • Modern Python integration: Uses async/await, dataclasses, Enums, and timezone-aware datetime/timedelta objects.
    • Type Safety: Full Mypy type checking support and relative imports.
    • Pydantic Support: Capability to generate Pydantic models (see documentation for specific generation steps).

    Note: This project is a reimplementation focused on idiomatic Python. While the wire format is identical to the official Google implementation, it is not a 1:1 drop-in replacement due to different method names and call patterns.

  3. Handle Protobuf oneof fields

    master

    Protobuf oneof groups ensure only one field in the group is set at a time.

    Accessing oneof fields:

    1. Pattern Matching (Python 3.10+): Use a match statement on the message instance for type-safe access.
    2. Utility Function: Use betterproto.which_one_of(message, group_name) to identify the set field. It returns a list containing [field_name, field_value], or ["", None] if no field is set.
    # Using match (Python 3.10+)
    match test:
        case Test(on=value):
            print(value)
        case Test(count=value):
            print(value)
        case Test(name=value):
            print(value)
    
    # Using which_one_of
    field_info = betterproto.which_one_of(test, "foo")
    # Returns e.g., ["on", True] or ["", None]
  4. Compile proto files with python-betterproto

    master

    To generate Python code from .proto files, use the protoc compiler with the --python_betterproto_out flag. You can invoke protoc directly or via grpcio-tools.

    Directly with protoc:

    mkdir lib
    protoc -I . --python_betterproto_out=lib example.proto

    Via grpcio-tools:

    pip install grpcio-tools
    python -m grpc_tools.protoc -I . --python_betterproto_out=lib example.proto
    mkdir lib
    protoc -I . --python_betterproto_out=lib example.proto
  5. Migrate from Google Protobuf to betterproto

    master

    betterproto is designed as a mostly 1-to-1 drop-in replacement for Google's official protocolbuffers package, but requires regenerating your protobufs.

    Key compatibility notes:

    • betterproto.Message.FromString is an alias for betterproto.Message.parse.
    • betterproto.Message.SerializeToString is an alias for betterproto.Message.__bytes__.
  6. Upgrade from v1.2.5 to v2.0.0b1

    master

    In version 2.0.0b1, generated code strictly follows the package structure defined in your .proto files. Files without a package will be combined into a single __init__.py. To avoid overwriting existing files, you should compile into a dedicated directory.

    Upgrade Steps:

    1. Remove your previously compiled .py files.
    2. Create a new empty directory (e.g., generated or lib/generated/proto).
    3. Regenerate your Python files into this new directory.
    4. Update your import statements to point to the new location (e.g., from generated import ExampleMessage).
  7. Install betterproto

    master

    You can install the core library via PyPI. If you need to compile .proto files, you must install the [compiler] extra to include the protoc plugin.

    Standard installation (runtime only):

    python3 -m pip install -U betterproto

    Windows installation (runtime only):

    py -3 -m pip install -U betterproto

    Installation with compiler plugin:

    python3 -m pip install -U "betterproto[compiler]"
    python3 -m pip install -U "betterproto[compiler]"
  8. Implement Async gRPC Clients and Servers

    master

    The project generates async gRPC stubs compatible with grpclib.

    Client Implementation: Use the generated *Stub class. Pass a grpclib.client.Channel to the stub. Remember to close the channel when finished.

    Server Implementation: Subclass the generated *Base class (e.g., EchoBase) and override the service methods. Use grpclib.server.Server to run the service.

    Note: Async gRPC stub generation is enabled by default and provides improved static type checking and code completion.

    # Client Example
    import asyncio
    import echo
    from grpclib.client import Channel
    
    async def main():
        channel = Channel(host="127.0.0.1", port=50051)
        service = echo.EchoStub(channel)
        response = await service.echo(echo.EchoRequest(value="hello", extra_times=1))
        print(response)
        channel.close()
    
    # Server Example
    import asyncio
    from echo import EchoBase, EchoRequest, EchoResponse, EchoStreamResponse
    from grpclib.server import Server
    from typing import AsyncIterator
    
    class EchoService(EchoBase):
        async def echo(self, echo_request: "EchoRequest") -> "EchoResponse":
            return EchoResponse([echo_request.value for _ in range(echo_request.extra_times)])
    
        async def echo_stream(self, echo_request: "EchoRequest") -> AsyncIterator["EchoStreamResponse"]:
            for _ in range(echo_request.extra_times):
                yield EchoStreamResponse(echo_request.value)
    
    async def main():
        server = Server([EchoService()])
        await server.start("127.0.0.1", 50051)
        await server.wait_closed()
  9. Run tests

    master

    Tests are managed via poethepoet. You can run standard tests or the full CI-style test suite.

    Standard Test Workflow:

    1. Generate assets from sample .proto files:
      poe generate
    2. Run the tests:
      poe test

    Full CI Test Suite: To run tests exactly as they are run in CI (using tox), use:

    poe full-test
    # Generate assets from sample .proto files required by the tests
    poe generate
    # Run the tests
    poe test
  10. Implement Async gRPC Clients

    master

    The generated code includes grpclib-based stubs for RPC services. To use a client, create a grpclib.client.Channel, pass it to the generated Stub class, and await the service methods. Important: Always close the channel when finished.

    import asyncio
    from grpclib.client import Channel
    import echo
    
    async def main():
        channel = Channel(host="127.0.0.1", port=50051)
        service = echo.EchoStub(channel)
        
        # Unary call
        response = await service.echo(value="hello", extra_times=1)
        print(response)
    
        # Streaming call
        async for response in service.echo_stream(value="hello", extra_times=1):
            print(response)
    
        channel.close()
    
    asyncio.run(main())
  11. Install betterproto and the protoc compiler plugin

    master

    To use betterproto for generating Python code from .proto files, you must install both the library and the compiler dependencies using the [compiler] extra.

    To install the latest stable version with compiler support:

    pip install "betterproto[compiler]"

    To install only the library (required only to run the code generated by the compiler, not to perform the compilation itself):

    pip install betterproto

    If you need the latest beta version, use the --pre flag:

    pip install --pre betterproto
    # Install both the library and compiler
    pip install "betterproto[compiler]"
    
    # Install just the library (to use the generated code output)
    pip install betterproto
  12. Recompile Google Well-known Types

    master

    Betterproto includes pre-compiled versions of Google's well-known types in src/betterproto/lib/google. If you modify the plugin output format, you must regenerate these files.

    To force compilation of google.protobuf (which is normally skipped because they are pre-compiled), use the --custom_opt=INCLUDE_GOOGLE flag.

    Assuming your google.protobuf source files are in /usr/local/include, use the following protoc command:

    protoc \
        --plugin=protoc-gen-custom=src/betterproto/plugin/main.py \
        --custom_opt=INCLUDE_GOOGLE \
        --custom_out=src/betterproto/lib \
        -I /usr/local/include/ \
        /usr/local/include/google/protobuf/*.proto