hamba/avro Documentation

repository·main·Indexed 19 days ago

https://github.com/hamba/avro

A high-performance Avro codec for Go used for encoding and decoding Go data structures to and from Avro binary format. It includes the avrogen tool for generating Go structs from Avro schemas (supporting Schema Registry integration) and the avrosv utility for schema validation.

Tokens
15.3K
Snippets
75
Records
87
Agent score
60%

What's inside hamba/avro

  1. How to handle Avro Unions

    main

    Avro unions can be handled in three ways in Go:

    1. map[string]any: If the union value is nil, a nil map is used. For non-nil values, a single key is decoded where the key is the Avro type name or schema full name.
    2. *T (Nullable Unions): For unions like ["null", "string"], you can use a pointer *T where T matches the non-null type. Slices can also be used directly.
    3. UnionConverter interface: For type-safe handling, implement the UnionConverter interface. Note: The implementation must use pointer receivers.
    type UnionConverter interface {
        FromAny(payload any) error
        ToAny() (any, error)
    }
    1. any (interface{}): You can provide an interface, but named types, maps, and slices must be registered using the Register function. For arrays and maps, the schema type/name is appended as a postfix (e.g., "map:string").
    type UnionConverter interface {
        // FromAny payload decode into any of the mentioned types in the Union.
        FromAny(payload any) error
        // ToAny from the Union struct
        ToAny() (any, error)
    }
    
    // Example implementation
    type UnionRecord struct {
        Int  *int
        Test *TestRecord
    }
    
    func (u *UnionRecord) ToAny() (any, error) {
        if u.Int != nil {
            return u.Int, nil
        } else if u.Test != nil {
            return u.Test, nil
        }
        return nil, errors.New("no value to encode")
    }
    
    func (u *UnionRecord) FromAny(payload any) error {
        switch t := payload.(type) {
        case int:
            u.Int = &t
        case TestRecord:
            u.Test = &t
        default:
            return errors.New("unknown type during decode of union")
        }
        return nil
    }
  2. Install the avrogen struct generator

    main

    Use the avrogen command-line tool to generate Go structs from Avro schemas. This tool can be installed via go install using the cmd/avrogen package.

    go install github.com/hamba/avro/v2/cmd/avrogen@<version>
  3. Install the avrosv schema validator

    main

    The avrosv utility is a command-line tool used to validate Avro schemas. It is useful in CI/CD pipelines to ensure schema changes are valid. It leverages the library's parsing logic and returns a non-zero exit code if validation fails.

    go install github.com/hamba/avro/v2/cmd/avrosv@<version>
  4. Map custom logical types with avrogen

    main

    You can map Avro logical types to specific Go types during code generation using the -logical-type flag. The format is avroLogicalType,goType[,importPath].

    • To map to an external type: avroLogicalType,goType,importPath (e.g., mapping uuid to github.com/google/uuid.UUID).
    • To map to a built-in Go type: avroLogicalType,goType (e.g., mapping date to int32).
    • To use multiple mappings, specify the -logical-type flag multiple times.
    # Mapping to an external package
    avrogen -pkg avro -o bla.go -logical-type uuid,uuid.UUID,github.com/google/uuid in.avsc
    
    # Mapping to a built-in Go type
    avrogen -pkg avro -o bla.go -logical-type date,int32 in.avsc
  5. Protect against untrusted input with MaxByteSliceSize

    main

    To prevent memory exhaustion attacks from untrusted input, the Config.MaxByteSliceSize option restricts the maximum size of bytes and string types created by the Reader.

    • Default: 1MiB.
    • Disable: Set the value to a negative number to disable this limit.
  6. Core Avro Schema Interfaces

    main

    The avro package defines several interfaces to represent different aspects of an Avro schema.

    • Schema: The base interface for all schema types. It provides methods to retrieve the Type(), the canonical String() representation, the SHA256 Fingerprint(), and a custom FingerprintUsing(FingerprintType) algorithm.
    • LogicalSchema: Represents a schema with an associated Avro logical type (e.g., decimal, uuid, timestamp-millis).
    • PropertySchema: Allows accessing custom properties attached to a schema via Prop(string) any.
    • NamedSchema: Represents schemas that have a name, namespace, and aliases (e.g., Record, Enum, Fixed). It provides Name(), Namespace(), FullName(), and Aliases().
    • LogicalTypeSchema: A schema that can optionally contain a LogicalSchema via the Logical() method.
  7. Use SchemaCache for type resolution

    main

    The SchemaCache is used to store and retrieve parsed schemas by their full name. This is essential for resolving references to types defined in different parts of a schema or in different files.

    DefaultSchemaCache is provided as a global instance, but you can create your own &SchemaCache{} to manage isolated sets of schemas.

  8. Generate code from a Schema Registry

    main

    If you provide a -schemaregistry URL, avrogen will fetch schemas from the registry instead of local files. When using a registry, the schema arguments must follow the format subject:version or subject:latest.

    Example: To fetch the latest schema for the subject user-value from a local registry:

    avrogen -pkg mypkg -schemaregistry http://localhost:8081 user-value:latest
    avrogen -pkg mypkg -schemaregistry http://localhost:8081 user-value:latest
  9. Initialize a Confluent Schema Registry Client

    main

    Use NewClient to create a new *Client for interacting with a Confluent Schema Registry. You can customize the client using functional options like WithHTTPClient for custom HTTP configurations or WithBasicAuth for credential-protected registries.

    import (
    	"context"
    	"net/http"
    	"github.com/hamba/avro/v2/registry"
    )
    
    ctx := context.Background()
    client, err := registry.NewClient("http://localhost:8081", 
    	registry.WithBasicAuth("user", "pass"),
    )
    if err != nil {
    	// handle error
    }
    client, err := registry.NewClient("http://localhost:8081", 
    	registry.WithBasicAuth("user", "pass"),
    )
  10. Use avrogen to generate Go code from Avro schemas

    main

    The avrogen CLI tool generates Go structs and associated code from Avro schema files or schemas hosted in a Schema Registry. It supports custom package names, field tags, logical type mappings, and custom templates.

    Basic Usage: avrogen [options] schemas

    Where schemas can be one or more file paths to .avsc files or entries in the format subject:version (or subject:latest) if using a Schema Registry.

    avrogen -pkg mypackage -o generated.go schema.avsc
  11. Initialize the Avro API with Config

    main

    To use the hamba/avro library, you should create a Config object to customize behavior and then call .Freeze() to obtain an API instance. The API instance is immutable and provides the primary methods for encoding and decoding. If you do not need custom configuration, you can use the pre-configured DefaultConfig provided by the package.

    Common configuration options include:

    • TagKey: The struct tag key used for en/decoding (defaults to "avro").
    • MaxByteSliceSize: Limits the size of bytes or string types created by the Reader (defaults to 1 MiB). Set to a negative number to disable.
    • DisableCaching: If true, forces encoders and decoders to be rebuilt on every call.
    • UnionResolutionError: Determines if an error is returned when a type cannot be resolved during union decoding.
    import "github.com/hamba/avro/v2"
    
    // Using the default API
    api := avro.DefaultConfig
    
    // Creating a custom API
    customConfig := avro.Config{
        TagKey: "my_tag",
        MaxByteSliceSize: 2 * 1024 * 1024, // 2 MiB
    }
    api := customConfig.Freeze()