DOCSEARCHbyAlgolia
Submit Docs

franz-go

repository·master·Indexed 25 days ago

https://github.com/twmb/franz-go

A high-performance Kafka client for Go. It provides features including administrative operations via kadm or kmsg, consumer group lag monitoring, Dead Letter Queue (DLQ) patterns, concurrent partition consumption, and support for Kafka transactions and exactly-once semantics (EOS). The library includes plugins for OpenTelemetry tracing and metrics (kotel), Prometheus metrics, and specialized observability for VictoriaMetrics. It also supports SASL/TLS authentication, Avro serialization via Schema Registry, and an in-memory Kafka cluster (kfake) for testing.

Tokens
33K
Snippets
36
Records
223
Agent score
82%

What's inside franz-go

  1. Overview of krammar DSL and Code Generator

    master

    The krammar directory contains a custom Domain Specific Language (DSL) and code generator used to produce Go structs and encoding/decoding functions for the Kafka wire protocol.

    Workflow:

    1. The generator reads definition files from definitions/ and an enums file.
    2. It outputs the generated Go code to ../pkg/kmsg/generated.go.

    Definition files are ordered numerically (e.g., 00_produce, 01_fetch). Request and response pairs must be defined together, with the response immediately following its request.

  2. Use the kvictoria plugin for VictoriaMetrics metrics

    master

    The kvictoria package provides metrics for VictoriaMetrics using the VictoriaMetrics/metrics library via a kgo.Hook.

    Warning: This plugin uses a non-standard implementation of histograms. It is intended specifically for users of the VictoriaMetrics database. If you use this with any other time-series database, your histograms will be unusable. If you require standard histograms, use the kprom plugin instead.

  3. Understand KIP-939 (2PC) Workflow

    master

    KIP-939 allows an external transaction coordinator (such as a database, Flink/Spark job, or custom orchestrator) to manage the outcome of a Kafka transaction.

    The Workflow:

    1. Prepare: The producer "prepares" a transaction by flushing and freezing it. The broker returns a durable {producerID, epoch} token.
    2. Persist: The external system atomically persists this token along with its own commit decision.
    3. Resolve: After a potential crash or restart, the external system instructs the producer to either commit or abort using that specific token.

    Key Feature: Unlike standard transactions, a 2PC transaction is never auto-aborted on timeout by the broker, allowing it to remain in a prepared state until the external coordinator resolves it.

  4. Choose between kadm and kmsg for Admin Requests

    master

    The repository provides two ways to issue Kafka admin requests:

    1. kadm package: A high-level, opinionated package that abstracts low-level details into intuitive methods and types with helper functions. Use this for most administrative tasks.
    2. kmsg package: A low-level package that allows you to construct Kafka requests directly. Use this if kadm does not provide the specific control or API coverage you need.

    All Kafka requests and responses are supported via generated code in kmsg.

  6. Understand Share Groups (KIP-932)

    master

    A share group is a Kafka consumption model (Kafka 4.0+) designed for queue-style workloads where per-record latency is prioritized over per-partition ordering. Unlike classic consumer groups that assign whole partitions to a single consumer, share groups allow multiple consumers to read from the same partition concurrently using broker-side acquisition locks.

    Key Differences from Classic Consumer Groups

    FeatureClassic Consumer GroupShare Group
    AssignmentWhole partitions, one consumer per partitionSubset of partitions; partitions can be shared across many consumers
    PositionOne offset per partitionNo client-side offset; broker tracks per-record state
    Ack ModelBulk: commit the next offset to readPer-record: each record is accepted/released/rejected individually
    RedeliveryOnly on consumer failure + rebalanceAutomatic when the broker's acquisition lock expires
    OrderIn-partition orderBest-effort; release re-queues
    RPCsFetch / OffsetCommitShareFetch / ShareAcknowledge
    Group ProtocolClassic or KIP-848ShareGroupHeartbeat (KIP-932)

    In franz-go, the share consumer is implemented as a distinct type (shareConsumer) rather than a mode switch within the classic consumer.

  7. Use plugins with kgo.Client for metrics and logging

    master

    The franz-go library provides a set of plugins to integrate external libraries for metrics and logging with a kgo.Client.

    Metrics Plugins

    To collect metrics, use the corresponding plugin with the kgo.WithHooks option:

    • kgmetrics: Integrates with go-metrics.
    • kprom: Integrates with prometheus.
    • kvictoria: Integrates with victoria metrics.

    Logging Plugins

    To integrate external logging frameworks, use the corresponding plugin with the kgo.WithLogger option:

    • klogrus: Integrates with sirupsen/logrus.
    • kzap: Integrates with uber-go/zap.
    • kzerolog: Integrates with rs/zerolog.

    Detailed usage examples for each plugin can be found in the examples/hooks_and_logging directory of the repository.

  8. Use the kgo package for core Kafka functionality

    master
    The kgo package is the primary entry point for franz-go. It provides all core functionality required to interact with Kafka, including support for transactions, regex topic consuming, partitioning strategies, data loss detection, and closest replica fetching. Most standard Kafka client operations should be performed using this package.

  9. Understand the franz-go Architecture Overview

    master

    The franz-go client is structured around several core abstractions that manage the lifecycle of Kafka communication:

    • Client (client.go): The top-level owner created via NewClient. It manages the broker pool, the metadata loop, and optional producer/consumer components.
    • Sink (sink.go): One per broker. It acts as an 'outbox' for a specific broker, collecting records and sending them in batched produce requests.
    • Source (source.go): One per broker. It acts as an 'inbox' for a specific broker, issuing fetch requests and buffering results for polling.
    • recBuf (sink.go): One per topic-partition, owned by a sink. It buffers records before they are sent. These are migrated between sinks when partition leaders change.
    • cursor (source.go): One per topic-partition, owned by a source. It tracks consumption progress (offset, epoch). These are migrated between sources when partition leaders change.
    • broker (broker.go): Manages TCP connections to a Kafka broker. Each broker maintains up to five specialized connections (produce, fetch, group, slow, general) to prevent workload interference.

  11. Enable internal logging with WithLogger

    master

    You can enable internal logging in kgo using the WithLogger option. By default, logging is disabled.

    To implement logging, you must satisfy the Logger interface. For development, you can use the provided BasicLogger. For production, it is recommended to use a structured logger. The repository provides drop-in plugins for popular loggers:

    It is recommended to use the info logging level.

  12. Set default values for fields

    master

    Fields can have default values specified in parentheses after the type. These defaults are used by the generated Default() method and, in flexible encoding, fields equal to their default are not written to the wire.

    Supported types for defaults:

    • Numeric: int32(-1), int64(0), float64(3.14)
    • Booleans: bool(true)
    • Nullable types: nullable-string(null), nullable-bytes(null)
    • Arrays: [int32](null)
    LogAppendTime: int64(-1) // v2+
RebalanceTimeoutMillis: int32(-1) // v1+
FinalizedFeaturesEpoch: int64(-1) // tag 1