Silverback .NET Message Bus

repository·master·Indexed 42 days ago

https://github.com/beagle1984/silverback

A .NET message bus and broker integration library for building reliable event-driven architectures. It provides a consistent developer experience with first-class support for Apache Kafka and MQTT, featuring broker-aware APIs, transactional outbox patterns, and support for Confluent Schema Registry.

Tokens
62.9K
Snippets
152
Records
219
Agent score
53%

What's inside Silverback

  1. What is Silverback

    master

    Silverback is a message bus and broker integration library for .NET designed to simplify event-driven architectures and asynchronous messaging. It provides a consistent, high-level API across different brokers while remaining 'broker-aware' to leverage specific capabilities.

    Key characteristics include:

    • First-class support for Apache Kafka and MQTT.
    • Kafka-optimized features: Partition-based parallelism, keys/partitioning, tombstones, Schema Registry integration, idempotency, and transactions.
    • Reliability features: Transactional outbox, error policies, and storage-backed features.
    • Operational features: Structured logging, diagnostics, and distributed tracing via System.Diagnostics.
  2. Overview of Silverback

    master

    Silverback is a message bus and broker integration library for .NET designed for building event-driven architectures and asynchronous workflows. It provides first-class support for Apache Kafka and MQTT.

    Key characteristics include:

    • Broker-aware API: While providing a consistent API across different brokers, it surfaces broker-specific features (e.g., Kafka's partition-based parallelism, keys, tombstones, Schema Registry, idempotency, and transactions) where they are relevant.
    • Reliability: Includes support for transactional outbox patterns, error policies, and storage-backed features.
    • Operational Features: Built-in structured logging, diagnostics, and tracing.
    • Cross-cutting Concerns: Built-in support for headers, validation, encryption, chunking, and batching.
    • Testability: Provides in-memory broker mocks and end-to-end testing helpers.
  3. What is Kafka client-side offset storage?

    master

    Kafka client-side offset storage is an application pattern where Silverback stores Kafka consumer offsets in an external storage layer (like a database) instead of relying on the Kafka broker.

    This is primarily used to achieve exactly-once processing relative to your own database. By persisting both your business data changes and the Kafka offset marker within the same database transaction, you ensure that either both are committed or both are rolled back. This prevents common failure modes where data is processed but the offset isn't committed, or vice versa.

    Key behaviors:

    • Silverback repositions the consumer based on the offsets found in your chosen store.
    • You should disable commits to the Kafka broker using .DisableOffsetsCommit() to avoid dual-source-of-truth issues.
  4. How the Consumer Behaviors Pipeline works

    master

    In Silverback, every message consumed is processed through a pipeline of behaviors. These behaviors handle cross-cutting concerns like decryption, deserialization, and validation.

    Key mechanics:

    • Ordering: The execution order is determined by the SortIndex property of each behavior.
    • Continuity: Each behavior is responsible for calling the next behavior in the pipeline using the provided handler. If a behavior does not call the next handler, the pipeline is short-circuited, and subsequent behaviors will not execute.
    • Context: Behaviors interact with the message via the ConsumerPipelineContext.
  5. How the Producer Behaviors Pipeline works

    master

    When producing a message in Silverback, the message passes through a pipeline of behaviors. Each behavior is a class implementing the IProducerBehavior interface.

    Key concepts:

    • Execution Order: The order of execution is determined by the SortIndex property of each behavior.
    • Pipeline Flow: Each behavior is responsible for passing the message to the next step in the pipeline by calling the next handler provided in the HandleAsync method.
    • Modification: Behaviors can modify the message (e.g., adding headers, encrypting content, or serializing) or perform side effects (e.g., starting a diagnostic Activity) before passing it along.
  6. Configure Message Routing

    master

    Silverback routes messages based on their type by default. You can customize this behavior using several methods:

    1. Routing Function

    Use ProduceTo with a lambda to determine the destination topic based on the message content or the IOutboundEnvelope<T> (which provides access to headers and metadata).

    Kafka Example:

    .Produce<MyMessage>(endpoint => endpoint
        .ProduceTo(envelope => envelope.Headers.GetValue<int>("x-random") % 2 == 0 ? "my-even" : "my-odd"))

    MQTT Example:

    .Produce<MyMessage>(endpoint => endpoint
        .ProduceTo(envelope => envelope.Headers.GetValue<int>("x-random") % 2 == 0 ? "my/even" : "my/odd"))

    2. Format Strings

    You can use a format string in ProduceTo where variable parts are generated from the message.

    Kafka Example:

    .ProduceTo(
        "my-{0}",
        message => message.Id % 2 == 0 ? new object[] { "even" } : new object[] { "odd" }))

    3. Dedicated Resolver Class

    For complex logic, implement a resolver class that can be injected via DI.

    • For Kafka: Implement IKafkaProducerEndpointResolver<TMessage>.
    • For MQTT: Implement IMqttProducerEndpointResolver<TMessage>.

    Register it using .UseEndpointResolver<TResolver>() in your producer configuration.

    4. Dynamic Routing

    To select the destination at publish time instead of configuration, enable dynamic routing using .ProduceToDynamicTopic() in the configuration, and then set the destination on the envelope during publication.

    Kafka:

    await _publisher.WrapAndPublishAsync(
        new MyMessage { ... },
        envelope => envelope.SetKafkaDestinationTopic("my-topic"));

    MQTT:

    await _publisher.WrapAndPublishAsync(
        new MyMessage { ... },
        envelope => envelope.SetMqttDestinationTopic("my/topic"));
  7. Configure parallelism for Kafka and MQTT

    master

    Parallelism behavior differs between Kafka and MQTT:

    Kafka

    Parallelism is enabled by default. Messages from different partitions are processed concurrently, while messages within the same partition are processed sequentially.

    • Use .LimitParallelism(n) to set a maximum degree of parallelism.
    • Use .ProcessAllPartitionsTogether() to disable partition-based parallelism and process all partitions in a single stream.

    MQTT

    Processing is sequential by default. To enable parallel processing, use .EnableParallelProcessing(n) on the client configuration.

    // Kafka: Limit parallelism
    .AddConsumer("consumer1", consumer => consumer
        .Consume<MyMessage>("endpoint1", endpoint => endpoint
            .LimitParallelism(3)
            .ConsumeFrom("my-topic-1")))
    
    // MQTT: Enable parallel processing
    .AddClient("my-client", client => client
        .EnableParallelProcessing(10)
        .Consume<MyMessage>("endpoint1", endpoint => endpoint
            .ConsumeFrom("messages/topic1")))
  8. Implement custom pipelines using Behaviors

    master

    Behaviors allow you to implement custom pipelines for message processing, similar to ASP.NET middleware. They are used to handle cross-cutting concerns such as logging, validation, or tracing.

    To create a behavior, implement the IBehavior interface and its HandleAsync method. The HandleAsync method receives the message and a MessageHandler next delegate, which represents the next step in the pipeline. You must call await next(message) to continue the pipeline execution.

    public class TracingBehavior : IBehavior
    {
        private readonly ITracer _tracer;
    
        public TracingBehavior(ITracer tracer)
        {
            _tracer = tracer;
        }
        
        public async Task<IReadOnlyCollection<object?>> HandleAsync(
            object message,
            MessageHandler next)
        {
            try
            {
                _tracer.TraceProcessing(message);
                object result = await next(message);
                _tracer.TraceProcessed(message);
                return result;
            }
            catch (Exception ex)
            {
                _tracer.TraceError(message, ex);
                throw;
            }
        }
    }
  9. Define message models in Silverback

    master

    Messages in Silverback can be any POCO (Plain Old CLR Object) class. While not mandatory, it is highly recommended to use the Silverback.Core.Model package to implement specific interfaces that define the semantic meaning of the message. This improves code structure and readability.

    Available interfaces in Silverback.Core.Model:

    • IEvent: Fire-and-forget notifications (no response expected).
    • ICommand / ICommand<TResult>: Triggers actions in another service; typically consumed by a single subscriber and can return a value.
    • IQuery<TResult>: Represents a data request; always returns a result.
    • IIntegrationMessage: Identifies messages intended for external brokers (e.g., Kafka, MQTT). This includes IIntegrationEvent and IIntegrationCommand.
    public class SampleMessage
    {
        public string Content { get; set; }
    }
  10. How distributed tracing works with Silverback producers

    master

    Silverback provides transparent distributed tracing propagation when producing messages. It leverages the .NET Activity class to capture timing and correlation context.

    When a message is produced, Silverback automatically propagates the current trace context by setting the following headers to maintain compatibility with the W3C Trace Context specification:

    • traceparent: Contains the trace ID and parent span ID.
    • tracestate: Contains trace-specific information.
    • tracebaggage: Contains any baggage (key-value metadata) associated with the current activity.

    This allows downstream consumers to continue the trace, providing end-to-end observability across services.

  11. Configure Kafka partition processing and concurrency

    master

    By default, Silverback processes partitions in parallel (one queue per partition), ensuring that messages within the same partition are processed sequentially.

    To change this behavior:

    • Use LimitParallelism to restrict the number of concurrent partitions being processed.
    • Use ProcessAllPartitionsTogether() to disable partition-based parallelism and process all messages from all assigned partitions in a single stream.
    services.AddSilverback()
        .WithConnectionToMessageBroker(options => options.AddKafka())
        .AddKafkaClients(clients => clients
            .WithBootstrapServers("PLAINTEXT://localhost:9092")
            .AddConsumer("consumer1", consumer => consumer
                .Consume<MyMessage>("endpoint1", endpoint => endpoint
                    .ProcessAllPartitionsTogether()
                    .ConsumeFrom("my-topic"))));