Spring Cloud Stream

repository·main·Indexed 21 days ago

https://github.com/spring-cloud/spring-cloud-stream

A framework for building message-driven microservice applications that provides an abstraction layer (binders) over message brokers. It supports various implementations including RabbitMQ, Apache Kafka, Amazon Kinesis, Google PubSub, Solace PubSub+, Azure Event Hubs, Azure Service Bus, and Apache RocketMQ. The framework leverages Spring Boot and Spring Integration to provide a unified programming model for defining message flows and managing connectivity to messaging middleware.

Tokens
105K
Snippets
279
Records
386
Agent score
75%

What's inside Spring Cloud Stream

  1. Overview of Spring Cloud Stream Release Train

    main

    Spring Cloud Stream is a release train that provides a unified programming model for building message-driven microservices. It consists of the Spring Cloud Stream core framework and various binder implementations that allow applications to connect to different message brokers.

    The release train includes:

    • Spring Cloud Stream Core: The central framework for defining message flows.
    • Curated Dependencies: The spring-cloud-stream-dependencies project manages a consistent set of compatible library versions.
    • Binder Implementations: Adapters that connect the core framework to specific messaging middleware.
  2. What is Spring Cloud Stream?

    main

    Spring Cloud Stream is a framework designed for building message-driven microservice applications. It builds upon Spring Boot to create standalone, production-grade applications and leverages Spring Integration to provide connectivity to various message brokers.

    Key concepts include:

    • Binders: Opinionated configurations of middleware from different vendors that provide persistent publish-subscribe semantics, consumer groups, and partitions.
    • Message Channels: The standard mechanism used by most binders (like Kafka and RabbitMQ).
    • Native Types: Specialized binders, such as the Kafka Streams binder, can bypass message channels to use native types like KStream and KTable directly.
  3. Introduction to Spring Cloud Stream

    main

    Spring Cloud Stream is a framework designed for building message-driven microservice applications. It leverages Spring Boot to create standalone applications and uses Spring Integration to provide connectivity to various message brokers.

    Key features include:

    • Opinionated configuration of middleware from multiple vendors.
    • Support for persistent publish-subscribe semantics.
    • Support for consumer groups and partitions.
    • Connectivity to message brokers via 'binders'.
    • Functional programming model where business logic is implemented using java.util.function.Function.
  4. Available Binder Implementations

    main

    Spring Cloud Stream supports various binder implementations that allow you to connect your application to different messaging middleware. Binders are responsible for the actual interaction with the message broker (e.g., creating topics/queues, producing, and consuming messages).

    Available implementations include:

    • RabbitMQ: spring-cloud-stream-binder-rabbit
    • Apache Kafka: spring-cloud-stream-binder-kafka
    • Amazon Kinesis: spring-cloud-stream-binder-aws-kinesis
    • Google PubSub (partner maintained): spring-cloud-gcp-pubsub-stream-binder
    • Solace PubSub+ (partner maintained): solace-spring-cloud-stream-starter
    • Azure Event Hubs (partner maintained): spring-cloud-stream-binder-for-azure-event-hubs
    • Azure Service Bus (partner maintained): spring-cloud-stream-binder-for-azure-service-bus
    • Apache RocketMQ (partner maintained): spring-cloud-alibaba/RocketMQ
  5. Use the Kafka Streams binder for native Kafka Streams integration

    main

    Spring Cloud Stream provides a dedicated binder implementation for Apache Kafka Streams. This allows a Spring Cloud Stream "processor" application to leverage native Apache Kafka Streams APIs directly within its core business logic. The binder is built upon the Spring for Apache Kafka project and provides binding capabilities for the three primary Kafka Streams types:

    • KStream
    • KTable
    • GlobalKTable

    Typical usage involves a processor application that reads records from an inbound topic, applies business logic using Kafka Streams APIs, and writes the results to an outbound topic. You can also define processor applications that do not have an outbound destination.

  6. What is the Reactive Kafka Binder?

    main

    The reactive Kafka binder is a dedicated binder based on the Reactor Kafka project. It was designed to enable full end-to-end reactive capabilities, such as backpressure and reactive streams, for applications using Apache Kafka.

    When to use it (Legacy): Previously, it was recommended for applications written using reactive types (Flux, Mono, etc.). However, due to its deprecation, users should now favor the standard Kafka binder with direct Project Reactor integration.

  7. Handle more than two inputs using Curried Functions

    main

    To support three or more input bindings, use currying (chaining partial functions). Each level of the function represents a new input binding.

    Binding Naming Convention: For a function f(x, y, z) implemented as x -> y -> z -> result:

    • Input 1: <name>-in-0
    • Input 2: <name>-in-1
    • Input 3: <name>-in-2
    • Output: <name>-out-0

    Example:

    @Bean
    public Function<KStream<Long, Order>,
            Function<GlobalKTable<Long, Customer>,
                    Function<GlobalKTable<Long, Product>, KStream<Long, EnrichedOrder>>>> enrichOrder() {
    
    return orders -> (
                  customers -> (
                        products -> (
                            orders.join(customers, (orderId, order) -> order.getCustomerId(), (order, customer) -> new CustomerOrder(customer, order))
                                    .join(products, (orderId, customerOrder) -> customerOrder.productId(), (customerOrder, product) -> {
                                        EnrichedOrder enrichedOrder = new EnrichedOrder();
                                        enrichedOrder.setProduct(product);
                                        enrichedOrder.setCustomer(customerOrder.customer);
                                        enrichedOrder.setOrder(customerOrder.order);
                                        return enrichedOrder;
                                    })
                            )
                    )
                );
    }
    @Bean
    public Function<KStream<Long, Order>,
            Function<GlobalKTable<Long, Customer>,
                    Function<GlobalKTable<Long, Product>, KStream<Long, EnrichedOrder>>>> enrichOrder() {
    
    return orders -> (
                  customers -> (
                        products -> (
                            orders.join(customers,
                                (orderId, order) -> order.getCustomerId(),
                                    (order, customer) -> new CustomerOrder(customer, order))
                                    .join(products,
                                            (orderId, customerOrder) -> customerOrder
                                                    .productId(),
                                            (customerOrder, product) -> {
                                                EnrichedOrder enrichedOrder = new EnrichedOrder();
                                                enrichedOrder.setProduct(product);
                                                enrichedOrder.setCustomer(customerOrder.customer);
                                                enrichedOrder.setOrder(customerOrder.order);
                                                return enrichedOrder;
                                            })
                            )
                    )
                );
    }
  8. Configure binder-specific environments and properties

    main

    Each named binder configuration can have its own isolated environment block. This block allows you to set any Spring Boot property specifically for that binder instance.

    Common use cases include:

    • Setting broker-specific connection details (e.g., spring.rabbitmq.host).
    • Activating specific Spring profiles for a single binder using spring.profiles.active.
    • Adding additional configuration classes via spring.main.sources to override or augment auto-configured beans for that specific binder.
    # Example: Adding a custom configuration class to a binder's environment
    environment:
        spring:
            main:
               sources: com.acme.config.MyCustomBinderConfiguration
    
    # Example: Activating a specific profile for a binder
    environment:
        spring:
            profiles:
               active: myBinderProfile
  9. How Producers and Consumers interact with destinations

    main

    Spring Cloud Stream distinguishes between producers and consumers through their relationship with binding destinations:

    Producers

    A producer sends messages to a binding destination. When using the bindProducer() method, you specify:

    1. The destination name within the external broker.
    2. The local destination instance name.
    3. Properties (such as partition key expressions) used by the adapter for that specific binding.

    Consumers

    A consumer receives messages from a binding destination. When using the bindConsumer() method, you specify:

    1. The destination name.
    2. A group name representing a logical group of consumers.

    Messaging Semantics

    • Publish-Subscribe: Each unique consumer group receives a copy of every message sent to the destination.
    • Queueing (Load Balancing): If multiple consumer instances share the same group name, messages are load-balanced across them. Each message is processed by exactly one instance within that group.
  10. Understand the Spring Cloud Stream Application Model

    main

    A Spring Cloud Stream application is built around a middleware-neutral core. It uses bindings to connect your application's input/output arguments to external message broker destinations.

    To bridge the gap between your code and a specific broker (like Kafka or RabbitMQ), Spring Cloud Stream uses Binder implementations. The Binder handles all the broker-specific details required to establish these bindings, allowing your application logic to remain decoupled from the underlying messaging infrastructure.

  11. Monitor Kafka consumer lag with Kafka binder metrics

    main

    The Kafka binder module exposes consumer lag metrics via Micrometer. The primary metric is spring.cloud.stream.binder.kafka.offset, which indicates the number of messages that have not yet been consumed from a specific binder topic by a specific consumer group.

    This metric includes the consumer group information, the topic name, and the actual lag (the difference between the latest offset on the topic and the committed offset). It is highly recommended for providing auto-scaling feedback to PaaS platforms.

    spring.cloud.stream.binder.kafka.offset
  12. How Spring Cloud Stream RabbitMQ provisions destinations by default

    main

    By default, the RabbitMQ binder automatically provisions exchanges and queues based on your binding configuration:

    • Exchanges: A topic exchange is created with a name derived from <prefix><destination>. If no destination is provided, it defaults to the binding name.
    • Consumer Queues:
      • If a group is specified: A queue is provisioned with the name <prefix><destination>.<group>.
      • If no group is specified: An anonymous, auto-delete queue is created.
    • Routing:
      • For non-partitioned bindings: The queue is bound to the exchange using the "match-all" wildcard routing key (#).
      • For partitioned bindings: The queue is bound using <destination>-<instanceIndex>.
    • Prefixes: The prefix is an empty String by default.
    • Required Groups: If an output binding specifies requiredGroups, a queue and binding are provisioned for each group listed.