EMQX Documentation

EMQX provides file transfer capabilities built directly on top of the standard MQTT protocol. Instead of using hybrid approaches (like combining MQTT for signaling and HTTP/FTP for content), EMQX implements a simple application-level protocol on top of MQTT. This allows devices to transfer diverse data types—such as audio/video, images, and diagnostic logs—using their existing MQTT connections and security frameworks.

Key capabilities include:

• Unified Channel: Uses the same MQTT connection as regular business operations, leveraging existing device authentication and authorization.
• Chunked Transfers: Enables lightweight clients to transfer large files, supporting files up to and exceeding the standard MQTT size limit (up to 256MB and beyond via chunking).
• Resumable Transfers: Supports pausing transfers to prioritize other data or recovering from network interruptions.
• Reliability: Uses QoS 1 level transmission combined with checksums and retransmission mechanisms to ensure 100% file integrity.
• Flexible Storage: Files can be saved to a local directory or S3-compatible object storage.

The EMQX user guides provide instructions for administrators and operators to manage, configure, secure, monitor, and maintain EMQX deployments. Key operational areas include:

• Cluster Management: Creating and managing clusters, including architecture, security, load balancing, node evacuation, and performance tuning.
• Configuration: Managing the configuration file and detailed configuration options.
• Multi-tenancy (Namespace): Logically grouping MQTT clients and managing tenant-level isolation, quotas, and rate limits.
• Management Interfaces: Using the REST API for HTTP-based management of clients, topics, and subscriptions; the Command Line Interface (CLI) for startup and management; and the EMQX Dashboard for a visual management console.
• Security: Configuring network/TLS settings, authentication, authorization, identity governance, and audit logs.
• Data Durability: Configuring MQTT Durable Sessions and data replicas for high availability.
• Observability: Monitoring metrics and system logs for debugging and system health.
• Extensibility: Extending EMQX using Hooks (including gRPC hooks) and developing/managing Plugins.

EMQX integrates with Apache Kafka (or Confluent) to bridge the gap between lightweight MQTT IoT devices and high-throughput backend streaming platforms. This integration allows for seamless data streaming where MQTT messages are ingested into Kafka topics for real-time processing, storage, and analysis, and data from Kafka topics can be delivered back to MQTT clients to trigger actions.

Key capabilities include:

• Bidirectional Messaging: Reliable transfer of messages from MQTT to Kafka (Sink) and Kafka to MQTT (Source).
• Payload Transformation: Use EMQX SQL rules to filter, enhance, or transform payloads before they reach Kafka.
• Flexible Topic Mapping: Map MQTT topics/user properties to Kafka topics/headers using 1:1, 1:N, or wildcard-based mapping.
• Partitioning Strategies: Route messages to specific Kafka partitions based on MQTT topics or client IDs.
• High Throughput: Support for both synchronous and asynchronous Kafka writes to balance latency and throughput.
• Observability: Access runtime metrics such as total message counts, success/failure rates, and current rates for each Sink and Source.

EMQX provides a ready-to-use integration with Snowflake to facilitate IoT data warehousing and analytics. EMQX acts as the connectivity layer, while Snowflake serves as the storage and processing platform. This integration is achieved using the EMQX Rule Engine and a Snowflake Sink.

Workflow Overview

1. Device Connection: Devices connect to EMQX via MQTT, triggering online events.
2. Message Publishing: Devices publish telemetry and status data to specific topics.
3. Rule Engine Processing: EMQX rules match topics and perform data transformation, filtering, or enrichment.
4. Writing to Snowflake: The rule triggers a Sink to write data using one of two modes:
   • Aggregated Mode: Batches messages into files and loads them via Snowflake Stage and Pipe.
   • Streaming Mode: Streams data directly using the Snowpipe Streaming API.

Use Cases

• Data Archiving: Long-term, compliant storage of IoT data.
• Data Analytics: Real-time or batch analysis for predictive maintenance and operational insights.

The emqx_bridge_mqtt_dq plugin forwards local MQTT messages to a remote MQTT broker using a disk-based buffer for resilience. It provides automatic retries when the remote broker is unavailable and supports topic rewriting.

Key Features

• Disk Buffering: Per-bridge disk queue partitions to prevent data loss during network outages.
• Automatic Retries: Continues attempting to publish if the remote broker is unreachable.
• Topic Rewriting: Use ${topic} in remote_topic to transform the destination topic.
• Resilience: If a queue partition exceeds queue.max_total_bytes, the oldest records in that partition are dropped to protect disk space.

How It Works

1. Local publishes are matched against a bridge's filter_topic.
2. Matching messages are appended to disk queue partitions.
3. Messages are published to the remote broker.
4. If publishing fails due to connectivity, the plugin retries automatically.

bridges {
  to-cloud {
    enable = true
    remote = cloud
    filter_topic = "devices/#"
    remote_topic = "fwd/${topic}"
    queue {
      max_total_bytes = "1GB"
    }
  }
}

remotes {
  cloud {
    server = "cloud-broker.example.com:8883"
    username = "bridge_user"
    password = "secret"
  }
}