DOCSEARCHbyAlgolia
Submit Docs

Sarama: Go Client Library for Apache Kafka

repository·main·Indexed 11 days ago

https://github.com/ibm/sarama

A high-performance, MIT-licensed Go client library for Apache Kafka providing robust producer and consumer capabilities. It includes support for SyncProducer, AsyncProducer, transactional producers for exactly-once semantics, and ClusterAdmin for partition reassignments. The library also provides a mocks subpackage for dependency injection in tests and supports SASL/SCRAM authentication.

Tokens
86.7K
Snippets
313
Records
444
Agent score
93%

What's inside Sarama

  1. Overview of Sarama command-line tools

    main

    The Sarama tools repository provides lightweight applications for exploring and instrumenting your Kafka cluster. Unlike standard Kafka tools, these are written in Go and do not require a JVM installation to function.

    Available tools include:

    • kafka-console-producer: Produces a single message to your Kafka cluster.
    • kafka-console-consumer: Consumes arbitrary partitions of a topic on your Kafka cluster.
    • kafka-producer-performance: Performs performance testing for both synchronous and asynchronous producers.
    • kafka-console-partitionconsumer: (Deprecated) Consumes a single partition of a topic.

  2. Implement a load-aware sticky consumer strategy

    main

    To implement a consumer strategy that accounts for member load (such as CPU usage, in-flight requests, or lag), you can implement the sarama.SubscriptionUserDataBalanceStrategy interface.

    A common pattern is to create a wrapper around the built-in sticky assignor. This wrapper can inject fresh load metrics (e.g., LoadSample containing CPU percent, in-flight count, and lag) into the member's JoinGroup subscription metadata during every rebalance cycle.

    While this example focuses on the metadata injection, a full production implementation would also:

    1. Implement the Plan method.
    2. Decode each member's UserData on the leader to weight the partition assignment based on the reported load.
    // The strategy is wired in via Config.Consumer.Group.Rebalance.GroupStrategies
config.Consumer.Group.Rebalance.GroupStrategies = []sarama.BalanceStrategy{
    &LoadAwareSticky{ /* ... */ },
}

  3. Configure keys and partitioning in kafka-console-producer

    main

    You can specify a message key using the -key flag. The tool follows a default partitioning logic:

    1. Manual partitioning: if -partition is provided.
    2. Hash partitioning: if -key is provided.
    3. Random partitioning: if neither is provided.

    You can override this behavior using the -partitioner argument.

    # Specify a key (uses hash partitioning by default)
echo "hello world" | kafka-console-producer -topic=test -key=key

# Override partitioning strategy to random
echo "hello world" | kafka-console-producer -topic=test -key=key -partitioner=random

  4. Ensure transactional-id uniqueness with ProducerProvider

    main
    When using transactional producers, each producer must have a unique transactional-id. This example demonstrates implementing a ProducerProvider pattern to manage this uniqueness. The ProducerProvider builds producers by appending a growing integer to the base transactional ID, ensuring that every instance created has a distinct identifier.

  5. Understand Sarama compatibility and API stability

    main

    Sarama follows semantic versioning and uses standard Go module version numbering to ensure API stability.

    Compatibility Guarantee: Sarama provides a "2 releases + 2 months" compatibility guarantee. This means the library officially supports:

    1. The two latest stable releases of Kafka.
    2. The two latest stable releases of Go.
    3. A two-month grace period for older releases.

    Note that older versions of Kafka may still work, but they are not officially guaranteed by the compatibility policy.

  6. Choose between SyncProducer and AsyncProducer for Kafka production

    main

    When building applications that produce messages to Kafka, choose your producer type based on your requirement for delivery confirmation:

    • SyncProducer: Use this if you must guarantee a message was successfully sent to the Kafka cluster before proceeding (e.g., before sending an HTTP response to a client). It blocks until the write is acknowledged.
    • AsyncProducer: Use this for 'fire and forget' scenarios where immediate confirmation is not required (e.g., background access logging). This allows you to respond to the client immediately while the message is produced in the background.

    Both SyncProducer and AsyncProducer are thread-safe. You can safely share a single producer instance across multiple concurrent goroutines (such as those managed by http.Server). Using a single shared producer is more efficient as it allows the producer to batch messages from concurrent requests together.

  7. Use sarama/mocks for dependency injection in tests

    main

    The sarama/mocks subpackage provides mock implementations of major Sarama interfaces. These mocks are designed to be used with dependency injection to test your Kafka-based applications without requiring a live Kafka cluster.

    To use them:

    1. Create the mock object (e.g., SyncProducer, AsyncProducer, or Consumer).
    2. Provide a *testing.T object during creation to allow the mock to report failures.
    3. Set expectations on the mock objects to define how they should behave during your test.
    4. Call Close() on the mocks at the end of your test. This triggers an expectation verification step that reports any unmet expectations to the provided *testing.T object.
    // Example conceptual workflow:
// 1. Create mock with *testing.T
// 2. Set expectations
// 3. Inject into your application
// 4. Close mock to verify expectations

  9. Alter partition reassignments to change replication factors

    main

    You can use ClusterAdmin.AlterPartitionReassignments to modify the replication factor of an existing topic. This process involves:

    1. Constructing a per-partition replica list.
    2. Calling AlterPartitionReassignments with the new configuration.
    3. Polling ListPartitionReassignments to monitor the progress until the reassignment is complete.

  11. Track partition reassignment progress

    main

    Calling AlterPartitionReassignments only submits the request; the actual data movement occurs in the background on the Kafka controller. To determine when the reassignment is finished, poll ListPartitionReassignments.

    An empty response for the specified topic in the returned status map indicates that the reassignment is complete.

    for {
    // Poll for status
    status, _ := admin.ListPartitionReassignments(topic, partitions)
    
    // If the topic key is missing or the slice is empty, movement is done
    if len(status[topic]) == 0 {
        break
    }
    time.Sleep(2 * time.Second)
}

  12. Get started with Sarama

    main

    Sarama is an MIT-licensed Go client library for Apache Kafka. To begin using the library, you can explore the following resources:

    • API Documentation: Detailed documentation and examples are hosted on pkg.go.dev.
    • Examples: Elaborate example applications are located in the ./examples directory of the repository.
    • Testing Mocks: If you need to mock Kafka for your unit tests, use the mocks subpackage.
    • CLI Tools: The ./tools directory contains command-line utilities for testing, diagnostics, and instrumentation.
    • FAQ: For common questions, refer to the Sarama Wiki FAQ.