LF Edge eKuiper Documentation

LF Edge eKuiper is a lightweight IoT data analytics and stream processing engine designed for resource-constrained edge devices. It provides a streaming software framework similar to Apache Flink, enabling users to create IoT edge analytics applications using either SQL-based or graph-based (similar to Node-RED) rules.

Key characteristics:

• Lightweight: Core server package is ~4.5MB with a ~10MB memory footprint.
• Cross-platform: Supports X86 (AMD 32/64), ARM (32/64), and PPC across Linux distributions, OpenWrt, MacOS, and Docker.
• Extensible: Supports custom Source, Sink, and UDF functions (User Defined Functions) via Golang or Python.
• Management: Manage plugins, streams, and rules via CLI, REST API, or Kubernetes ConfigMaps. A web-based management dashboard is also available.

eKuiper can be used as the central stream processing engine in an edge-cloud collaboration architecture for IIoT, IoV, and Smart City applications. In this model, eKuiper is deployed at the edge to perform stream processing, rule engine execution, and message routing.

Data Flow Pattern:

1. Ingestion: eKuiper collects data from edge sources such as Neuron (IIoT connectivity server), NanoMQ (ultra-lightweight MQTT broker), or general HTTP services.
2. Processing: eKuiper performs real-time data processing and analysis.
3. Egress/Storage: eKuiper publishes processed results to a cloud-based MQTT broker like EMQX or saves data to local storage.

eKuiper uses a SQL-like query language to perform transformations and computations over streams of events. This allows you to process real-time IoT data using familiar SQL syntax. The language supports several core components:

• Stream specifications: Defining the source of the data.
• Query language elements: The basic building blocks of your queries.
• Windows: Performing computations over specific time or count intervals.
• Built-in functions: Using pre-defined operations for data manipulation.
• Extensions: Extending functionality via plugins or external service calls.

The eKuiper CLI (Command Line Interface) is a tool used to manage streams and rules within the eKuiper ecosystem. It acts as a client to the eKuiper server. While the eKuiper server is responsible for the engine execution (processing stream/rule definitions, managing rule states, and handling I/O), the CLI provides the interface for administrators and developers to interact with these components.

Key management areas include:

• Streams: Managing the data sources and flows.
• Rules: Managing the logic and queries applied to streams.

LF Edge eKuiper is an open-source, lightweight IoT edge data analytics and stream processing software designed to run on resource-constrained hardware. It provides a real-time stream computing framework similar to Apache Flink, allowing users to create edge analytics applications in minutes using SQL-based or Graph-based rules.

Key Capabilities:

• Data Processing: Extraction, transformation, filtering, sorting, grouping, aggregation, and joins.
• Functions: Over 60 built-in functions covering math, string, aggregation, and hash operations.
• Windowing: Supports 4 types of time windows and count windows.
• Extensibility: Users can extend the system via Golang or Python in three areas: Sources (data ingestion), Sinks (data output), and SQL Functions (custom logic like AI/ML calls).

eKuiper provides a Go native plugin system that allows users to extend the engine's capabilities by implementing specific interfaces. There are three primary types of extensions:

1. Source Extensions: Used to extend data stream sources (e.g., integrating data from custom message servers). eKuiper includes a built-in MQTT source.
2. Sink/Action Extensions: Used to extend data destinations (e.g., pushing data to custom databases, web interfaces, or file systems). eKuiper includes built-in MQTT and Log file sinks.
3. Function Extensions: Used to add custom SQL functions to the engine's SQL processing capabilities.

eKuiper is a lightweight stream processing engine designed for edge computing. It is applicable in several key domains:

• Industrial IoT (IIoT): Deploy eKuiper at the edge near data sources to perform real-time analysis and intelligent decision-making for smart manufacturing.
• Internet of Vehicles (IoV): Used in vehicle-to-cloud solutions (often combined with the QUIC protocol) to handle the explosive growth of data collection requirements in connected vehicles, ensuring easy deployment and portability.
• Public Data Analysis: Used to extract valuable information from large-scale public data sharing platforms.

eKuiper supports seamless integration with various IoT and edge computing platforms to extend its data processing capabilities. Key integration paths include:

• Edge-Cloud Collaboration: Building solutions for IIoT, IoV, and Smart Cities using eKuiper as the edge component.
• EdgeX Foundry: Using eKuiper as the real-time stream processing module within the EdgeX Foundry ecosystem.
• Neuron: Integrating with Neuron to perform computations on collected data without additional configuration, reducing resource requirements.
• KubeEdge: Deploying eKuiper instances in containers via KubeEdge to enable low-latency edge computing by subscribing to device data via MQTT.
• OpenYurt: Deploying eKuiper in OpenYurt clusters and utilizing yurt tunnels for cloud-to-edge management.