DOCSEARCHbyAlgolia
Submit Docs

NewLife.MQTT

repository·master·Indexed 40 days ago

https://github.com/newlifex/newlife.mqtt

A high-performance, zero-dependency MQTT protocol implementation for .NET providing both a client and a broker. Supports MQTT 3.1, 3.1.1, and 5.0 across multiple .NET runtimes. Features include support for TCP, SSL, WebSocket, QUIC, and Reliable UDP transports, automatic re-subscription, shared subscriptions, message bridging, and a rule engine. Includes enterprise-grade capabilities such as cluster deployment via ClusterServer and custom authentication via IMqttAuthenticator.

Tokens
24.3K
Snippets
35
Records
115
Agent score
100%

What's inside NewLife.MQTT

  1. Overview of NewLife.MQTT Modules

    master

    NewLife.MQTT is composed of several core functional modules that handle the full lifecycle of MQTT communication, from protocol encoding to large-scale clustering. The primary modules include:

    • PROT (Protocol Encoding/Decoding): Handles the core MQTT protocol logic.
    • TRAN (Transport Protocol): Manages the underlying transport layers, including TCP, WebSockets, and advanced protocols like QUIC and Reliable UDP.
    • CLNT (Client): The client-side implementation for connecting to brokers.
    • SRV (Server): The broker/server-side implementation.
    • ENT (Enterprise Extensions): Provides advanced features like rule engines and WebHooks.
    • CLUS (Cluster): Manages distributed node operations, including node management, subscription synchronization, message forwarding, session migration, Raft-based consistency, and large-scale topic routing.
    • SEC (Security Enhancement): Handles security-related features.

  2. Overview of NewLife.MQTT capabilities

    master

    NewLife.MQTT is an IoT messaging protocol implementation providing both MqttClient and MqttServer capabilities.

    Key Features:

    • MQTT over QUIC: Rare industry support for high-performance transport.
    • MQTT over Reliable UDP: Original implementation for efficient communication.
    • Automatic Re-subscription: Automatically restores subscriptions after a connection loss.
    • Enterprise Features: Built-in enterprise-grade features provided for free.
    • Cloud Compatibility: The client supports connection to Alibaba Cloud IoT.

  3. Understand the NewLife.MQTT product ecosystem

    master

    NewLife.MQTT is a complete, zero-third-party-dependency MQTT implementation for .NET. It is designed for IoT developers and enterprise integrators, providing a one-stop solution from protocol encoding to enterprise-grade brokers. The ecosystem is divided into several key components:

    • NewLife.MQTT (NuGet package): The core library containing protocol encoding/decoding, the MqttClient, MqttServer, clustering, bridging, and the rules engine.
    • NewLife.MqttServer: A ready-to-use MQTT Broker application that integrates with the Stardust registry.
    • NewLife.MQTT.Storage: A persistence extension that allows you to implement IStorageProvider to persist messages and sessions using Redis or databases.

  4. Understand the SRV Server Architecture

    master

    The SRV Server is a high-performance MQTT Broker designed for platform operations. It is composed of several specialized modules that handle networking, message routing, connection lifecycle, and security.

    Core Module Responsibilities

    ResponsibilityCore TypeDescription
    Network ServiceMqttServer (inherits NetServer)Listens on TCP/UDP/IPv6 (default port 1883).
    Message ExchangeMqttExchange (IMqttExchange)A global singleton managing subscription routing, Retain messages, persistent sessions, and $SYS topics.
    Connection HandlingMqttHandler (IMqttHandler)A transient instance per connection that manages the MQTT packet pipeline.
    Security & AuthIMqttAuthenticatorPluggable authentication (Username/Password + OAuth tokens) and ACL.
    Session CapabilitiesMqttSessionCapabilitiesManages MQTT 5.0 session capability negotiation and runtime management.
    PersistenceIStorageProviderEnables recovery of messages and sessions across restarts (e.g., via DbExchange or RedisExchange).
    HTTP ManagementMqttManagementServerProvides RESTful APIs and a MqttDashboard for monitoring.

  5. Explore the NewLife.MQTT project composition and target frameworks

    master

    The project is distributed through several components:

    • NewLife.MQTT: The core NuGet package containing protocol codec, client, server, cluster, bridging, and rule engine logic. This is the primary package for integrated delivery.
    • NewLife.MqttServer: A standalone Broker application that integrates with the Stardust registry.
    • NewLife.MQTT.Storage: An extension for persistence, providing the IStorageProvider interface for Redis or database backends.

    Supported Target Frameworks: net45, net461, netstandard2.0, netstandard2.1, net6.0, net7.0, net8.0, net9.0, net10.0.

  6. Understand the scope and limitations of NewLife.MQTT

    master

    Before integrating NewLife.MQTT, be aware of the following architectural decisions and limitations:

    • Management UI: The Dashboard is an embedded WebPanel for the local machine (SRV-8) and does not provide a centralized UI for managing multiple distributed instances.
    • Database Persistence: The core library does not bundle a specific database. Persistence must be implemented by extending the IStorageProvider interface (related to SRV-7 / CLUS-6).
    • Authentication: While OAuth is built-in (SRV-3), LDAP authentication is not provided out-of-the-box. You must implement it by extending the IMqttAuthenticator interface.
    • Runtime Compatibility:
      • Minimum requirement is .NET Framework 4.5.
      • Supports a wide range from net45 to net10.0.
      • Does NOT support .NET Framework 4.0 or below.
      • Does NOT support .NET AOT/NativeAOT (priority is given to full framework compatibility).
    • Protocol Support:
      • MQTT 6.0 is not supported.
      • QUIC Server-side listening is currently under evaluation and depends on the stability of .NET 9+ QUIC APIs.

  7. Understand NewLife.MQTT performance characteristics

    master

    NewLife.MQTT is a full-featured MQTT server and client component supporting TCP, WebSocket, QUIC, and UDP protocols, with support for MQTT V3.1.1 and V5.0.

    Key Performance Findings:

    • QoS0 Peak Throughput: Approximately 53,800 msg/s (using 5000-message batches).
    • QoS1 Peak Throughput: Approximately 20,200 msg/s.
    • Bottlenecks: Performance is primarily limited by topic matching (full table scans) and per-message heap allocations, rather than message encoding/decoding.
    • Encoding/Decoding: The encoding/decoding layer operates at nanosecond speeds and does not constitute a performance bottleneck.

  8. Overview of CLUS Cluster Component Responsibilities

    master

    The cluster is organized into functional layers. Developers extending or configuring the cluster should be aware of these layers:

    LayerKey ComponentsResponsibility
    ConsensusRaftClusterRaft consensus for subscription metadata (Leader election, Log replication, Snapshots).
    PropagationGossipExchangeOptional Gossip-based propagation for large-scale scenarios (Status: Pending).
    ShardingTopicShardManager, TopicRouterConsistent hashing for topic sharding and shard-aware routing (v5.x core).
    AccelerationBloomFilterExchange, HotTopicDetectorBloom filter pre-checks and hot-topic detection with dynamic replica adjustment.
    ManagementClusterServer, ClusterNodeNode lifecycle: Join, Ping, Leave, and timeout removal.
    TopologyHierarchicalClusterManagement of multi-tier cluster topologies (Edge/Regional/Core).
    DiscoveryClusterDiscoveryUDP broadcast or SPI-based automatic node discovery.
    RoutingClusterExchangeMapping subscription relationships and cross-node message forwarding.
    OptimizationBatchForwardPipe, ConnectionAggregation, etc.Batch forwarding, connection aggregation, subscription delta sync, and route degradation.
    StorageMqttStorageExchangePersistence via Redis or Databases (via NewLife.MQTT.Storage).

  9. Understand the CLUS Cluster Architecture

    master

    The CLUS implementation allows for horizontal scaling from 3 to 1000+ nodes using a Raft-based consensus engine.

    Core Cluster Concepts

    • Consistency: Uses the Raft consensus engine for strong consistency in subscription management (Leader election, Log replication, and Snapshots).
    • Node Discovery: Supports UDP broadcast for local networks and manual ClusterNodes configuration or SPI-based discovery (Stardust/Consul).
    • Message Routing: Uses a three-level routing decision: Topic $\rightarrow$ Shard $\rightarrow$ Node. This is optimized via a Consistent Hashing Sharding Engine (1024 shards, 256 virtual nodes per node, 3 replicas).
    • Scaling Topologies: Automatically switches between Flat ($<50$ nodes), Sharded ($50\text{--}1000$ nodes), and Multi-layer ($1000+$ nodes) architectures.
    • Efficiency Optimizations:
      • Bloom Filters: Uses MurmurHash3-based Bloom filters to reduce invalid message forwarding across nodes.
      • Batching Pipeline: Merges messages by target node with a $5 ext{ms}/64 ext{KB}$ threshold to reduce RPC calls by $\ge 100 imes$.
      • Connection Aggregation: Edge nodes can aggregate connections at a ratio of up to $1000:1$.

  10. Configure persistence for NewLife.MQTT

    master

    By default, NewLife.MQTT uses memory-level persistence, meaning sessions and messages will be lost if the service restarts. To ensure data durability across restarts, you must configure the NewLife.MQTT.Storage package to provide persistent storage.

    // Note: Requires additional configuration via NewLife.MQTT.Storage
// to prevent data loss on service restart.

  11. How Cluster Discovery works

    master

    Nodes can find each other automatically using UDP broadcasting.

    • Protocol: The broadcast format is MQTT-CLUSTER:{IP}:{Port} (UTF-8).
    • Port: The broadcast occurs on ClusterPort + 1.
    • Interval: Announcements are sent every 30 seconds.

    Users can extend discovery logic by implementing the IClusterDiscoveryProvider interface (e.g., for Consul or Stardust), though UdpDiscoveryProvider is the built-in default.

    public interface IClusterDiscoveryProvider
{
    String Name { get; }
    Task StartAsync(ClusterServer cluster, CancellationToken token);
    Task StopAsync(CancellationToken token);
}

  12. Use HierarchicalCluster for Large Scale Topologies

    master

    For massive deployments, HierarchicalCluster manages a three-tier topology to handle routing and connection aggregation efficiently:

    1. Edge Nodes: Handle direct device connections (TCP/TLS/WS). They maintain local subscriptions and report topic summaries (Bloom Filters) to the Regional tier if a local match is not found.
    2. Regional Nodes: Aggregate routing info from multiple Edge nodes and maintain regional topic summaries. They handle cross-Edge forwarding within the region.
    3. Core Nodes: Act as the global routing coordinators. They maintain regional Bloom Filters and handle cross-regional message forwarding.

    This hierarchy allows the system to scale from hundreds of nodes to thousands by reducing the complexity of the routing mesh.