KafkaFlow

KafkaFlow provides administrative infrastructure to inspect details about KafkaFlow consumers and perform administrative tasks. You can interact with this infrastructure using two primary methods:

1. Administration Web API: A programmatic interface for managing consumers and retrieving details.
2. Dashboard: A visual interface for monitoring and administration tasks.

If you need to control which worker handles a specific message, implement the IWorkerDistributionStrategy interface.

In v3, the method .GetWorkerAsync() receives a WorkerDistributionContext instead of just a partition key. This context provides more metadata, including the message topic and partition, allowing for more complex distribution logic.

A KafkaFlow consumer is a coordinated group of components that manage the lifecycle of message consumption. The flow of a message follows this path:

1. Kafka Consumer: Runs the Confluent Client in a background task to fetch messages from assigned topics/partitions.
2. Consumer Worker Pool: Orchestrates the creation/destruction of Workers and uses a Distribution Strategy to route messages.
3. Distribution Strategy: An algorithm that selects which Worker receives a message.
4. Workers: Parallel units responsible for processing messages. They use internal buffers to prevent idling.
5. Middlewares: A consumer-specific collection of processing steps (implementing IMessageMiddleware) shared among all workers in that consumer.
6. Offset Manager: Collects offsets from workers and orchestrates their storage in Kafka to prevent overrides during concurrent processing.

KafkaFlow is built on a middleware-oriented architecture. Messages travel through a pipeline of middlewares that are invoked in the sequence they are defined in your configuration.

Key Execution Rules:

• Order Matters: Middlewares are executed in the exact order they are registered.
• Scope: Every consumer and producer has its own unique set of middleware instances. However, for a single consumer, the middleware instances are shared across all workers of that consumer.
• Dependency Injection: Middlewares are instantiated by your configured DI container. You can inject any service registered in your container into the middleware's constructor.
• Lifetime Control: When registering middlewares, use the overloads of the Add<TMiddleware>(MiddlewareLifetime) method to control their lifecycle (e.g., Singleton vs Transient).

The middleware determines the message type using one of two primary methods depending on your serialization setup:

1. With Schema Registry: The schema is retrieved from the registry. The first 5 bytes of the message represent the SchemaId used for identification.
2. Without Schema Registry: The DefaultTypeResolver is used. It looks for a Message-Type header containing the fully qualified type name.

You can implement a custom type resolution strategy by implementing the IMessageTypeResolver interface and registering it via .AddSerializer or .AddDeserializer in your consumer/producer middleware.

To support dynamic worker scaling, KafkaFlow requires messages to be explicitly signaled as completed so that workers can be scaled down safely.

• If you were previously using WithManualStoreOffsets(), you must now use .WithManualMessageCompletion().
• To signal that a message is finished, use context.ConsumerContext.Complete() instead of context.ConsumerContext.StoreOffset().

By default, KafkaFlow uses AutoMessageCompletion. Manual completion is particularly useful in scenarios like Batch Consume where you do not want to complete the message immediately.

// Configuration change
.WithManualMessageCompletion()

// Usage in handler
context.ConsumerContext.Complete();

The Typed Handler Middleware enables the execution of specific handlers based on the incoming message type. This is the recommended approach when a topic contains multiple different message types. When a message arrives, the middleware identifies its type and invokes the corresponding IMessageHandler<T> implementation.

// Implementation of a handler
public class ProductCreatedHandler : IMessageHandler<ProductCreatedEvent>
{
    public Task Handle(IMessageContext context, ProductCreatedEvent message)
    {
        // Handle the message logic here
        return Task.CompletedTask;
    }
}

The Consumer Lag-Based Worker Balancer is a mechanism for dynamic resource allocation. Instead of a static worker count per instance, KafkaFlow monitors the lag in the Kafka topic.

Instances assigned to partitions with higher message lag are allocated more worker threads (up to the maxInstanceWorkers limit), while instances with lower lag receive fewer (down to the minInstanceWorkers limit). This ensures that the total pool of totalWorkers is used where it is most needed to reduce lag, preventing uneven workloads in elastic infrastructures.

KafkaFlow uses an Offset Manager to orchestrate offsets from multiple workers before committing them to Kafka. This prevents offset overrides during concurrent processing.

Critical Reliability Note: When an application stops, there is a high probability that messages have been processed and stored in the OffsetManager but have not yet been committed to Kafka. Upon restart, these messages will be processed again. Your application logic must be idempotent or prepared to handle duplicate message processing.