Apache RocketMQ

RocketMQ Proxy is a stateless component designed to enable stateless consumption behavior by leveraging the pop consumption mechanism. It acts as a traffic interface that translates incoming traffic (such as gRPC) into the customized Remoting protocol used to communicate with Broker and Namesrv.

Key responsibilities of the Proxy include:

• Protocol Translation: Converting external protocols to internal Remoting protocol.
• Security: Handling SSL, authorization, and authentication.
• Observability: Managing logging, tracing, and metrics.
• Traffic Management: Handling connection management and traffic governance.
• Message Support: Supports normal, fifo, transaction, and delay message types via pop consumption mode.

RocketMQ 5.0 introduces several advanced features:

• POP Consumption: Server-side rebalance and lightweight consumer client support.
• StaticTopic: Logic queue design for static topics.
• BatchConsumeQueue: Improved throughput for batch message processing.
• Automatic Master-Slave Switchover: Enhanced high availability.
• BrokerContainer: Containerized broker capabilities.
• SlaveActingMasterMode: A specific mode for slave nodes.
• gRPC Proxy: Support for gRPC-based proxy communication.

To understand the fundamental model and capabilities of Apache RocketMQ, refer to the following documentation:

• Concepts: Learn about the basic conceptual models used by RocketMQ.
• Features: Understand the functional capabilities and features implemented by RocketMQ.

Light Message Queue (LMQ) uses a read-amplification strategy to optimize storage and performance. Instead of writing multiple copies of a message for different consumption scenarios, a single message is written to the commitlog. The broker then dispatches multiple ConsumerQueue indices based on different requirements.

This allows a single message to support:

1. Server-side scenarios (MQ/AMQP): Traditional consumption based on top-level Topic queues.
2. Client-side scenarios (MQTT): Consumption based on multi-level MQTT topics (LMQ).

A RocketMQ-on-DLedger Group is a broker group designed for high availability using the Raft algorithm.

Key characteristics:

• Automatic Failover: The group automatically elects a Leader via Raft; other nodes act as Followers.
• Data Consistency: The Leader replicates data to Followers to maintain system consistency.
• Node Requirements: A group requires at least 3 nodes to tolerate one node failure. While 2 nodes can be deployed, they lack failover capabilities.
• Scaling: You can scale horizontally by deploying multiple independent DLedger Groups to provide external services.

In RocketMQ 5.x, BrokerContainer is a new deployment mode that allows multiple independent Brokers (Master, Slave, or DLedger) to run within a single process.

Key Concepts

• Resource Efficiency: Multiple brokers share the same process, improving single-node resource utilization.
• Shared Transport Layer: All brokers within a BrokerContainer share the same underlying transport layer (Netty instance, computing resources, and protocol stack), though each broker maintains its own port and ProcessorTable.
• High Availability: Enables "Peer Deployment" patterns (e.g., 2-replica or 3-replica configurations) where nodes are cross-deployed so that if one physical node fails, the remaining nodes contain both master and slave brokers, ensuring continuous message availability.
• Isolation: Each broker within the container has its own isolated CommitLog.

A RocketMQ-on-DLedger Group is a high-availability deployment pattern using the Raft consensus algorithm. It consists of a group of Brokers with the same name (at least 3 nodes).

Key characteristics:

• Automatic Failover: Raft automatically elects a Leader; other nodes act as Followers.
• Data Consistency: Data is replicated between the Leader and Followers to ensure high availability and consistency.
• Scalability: You can deploy multiple independent DLedger Groups to scale horizontally.

RocketMQ provides two types of message ordering to ensure messages are consumed in the order they were published:

1. Global Message Ordering: All messages within a specific Topic are published and consumed in strict First-In-First-Out (FIFO) order. Use this when performance requirements are low and strict global sequence is mandatory.
2. Partitioned Message Ordering: Messages within a Topic are partitioned using a sharding key. Messages sharing the same sharding key are sent to the same partition and consumed in strict FIFO order. This allows for high-performance parallel consumption across different partitions while maintaining order for specific groups (e.g., all messages for a specific orderId).

The DLedgerController ensures metadata consistency using the Raft protocol. It operates in two roles:

1. Active DLedgerController (Leader): The elected leader that accepts event requests from clients, initiates consensus through DLedger, and applies changes to an in-memory metadata state machine.
2. Not Active DLedgerController (Follower): Replicates event logs from the Active controller via DLedger and applies them to its own state machine.

To reduce IO load on a Master Broker (especially when a consumer needs to reset offsets to a significant time in the past), you can enable consumption from Slave Brokers.

Conditions for recommendation: When slaveReadEnable=true is set, the Master Broker will recommend consuming from a Slave Broker if the consumer's offset exceeds the accessMessageInMemoryMaxRatio (default is 40%).