DOCSEARCHbyAlgolia
Submit Docs

LF Edge eKuiper

website·Indexed 34 days ago

https://ekuiper.org/

A lightweight, SQL-based stream processing software designed for IoT edge computing. It provides capabilities for real-time data analysis, stream transformation, and integration with various sources and sinks including MQTT, Kafka, InfluxDB, and Redis. The platform supports extensibility via Go native plugins, Wasm plugins, and JavaScript script functions, and includes a management console for rule deployment and monitoring.

Tokens
223.9K
Snippets
1.4K
Records
1.8K
Agent score
68%

What's inside eKuiper

  1. Overview of eKuiper Script Functions

    Script functions provide a mechanism to quickly extend eKuiper's functionality without the need for compilation or packaging, reducing development and operation costs compared to plugin development. They are registered as plain text and can be exported/imported similarly to rules during version upgrades or node deployments. Currently, eKuiper supports JavaScript via the built-in Goja interpreter (ECMA 5.1 standard).

  2. Overview of eKuiper Connection Management

    eKuiper v2 uses an internal connection pool to manage complex resources, handle automatic reconnections, and provide connection status tracking. Supported connection types include MQTT, Neuron, EdgeX, SQL, HTTP (REST sink, HTTP Pull/Push source), WebSocket, and Kafka.

    Connections are managed in three ways:

    1. Rule-attached: Managed by the Source/Sink implementation; created when the rule starts and closed when it ends.
    2. Anonymous Pool Connections: Managed by the connection pool. The rule generates a unique resource ID. These are visible via the Connection API and are deleted when the rule is deleted.
    3. User-created Connections: Created via the Connection Management API with a unique ID. These are independent physical connections that start immediately and can be shared across multiple rules or sources/sinks.

  3. Overview of eKuiper External Functions

    eKuiper allows mapping external services to internal SQL functions via hot-updates. This enables SQL rules to call external services for data transformation or processing. There are two types of external functions:

    1. Schema-based: Requires a schema file (Protobuf) to describe the API interface.
    2. Schemaless: Applicable only to simple REST protocol scenarios.

  4. Overview of eKuiper Streams

    Streams in eKuiper are the operational form of data source connectors. They act as triggers for rules, where every incoming event triggers the computation within the associated rule. Streams must be unbounded when used in this capacity. eKuiper supports both schema-less streams (ideal for IoT/edge scenarios with flexible data) and schema-defined streams (which provide better validation and SQL optimization during compilation).

  5. Overview of eKuiper platform integrations

    eKuiper provides built-in integration capabilities to enable real-time stream processing across various edge computing ecosystems. Supported integrations include:

    • Edge Cloud: Processing data efficiently at the point of generation in edge cloud scenarios.
    • EdgeX Foundry: Enhancing the modular IoT edge computing platform with real-time stream processing.
    • Neuron: Pairing Neuron's device connection and data processing capabilities with eKuiper's stream processing.
    • KubeEdge: Extending Kubernetes-style orchestration to the edge for real-time processing.
    • OpenYurt: Streamlining the deployment and management process of eKuiper at the edge.

  6. Overview of eKuiper Portable Plugins

    Portable plugins are a supplement to native plugins, designed to provide the same functionality (custom sources, sinks, and functions) while running in a more general environment and supporting multiple programming languages. Unlike native plugins, portable plugins run as independent processes and communicate with the main eKuiper process using nanomsg. They support hot updates without restarting rules because the main process maintains the control and data channels even when the plugin process is updated.

  7. Overview of LF Edge eKuiper

    LF Edge eKuiper is a lightweight, open-source IoT edge analytics and stream processing software implemented in Golang. It is designed to run on resource-constrained edge devices and provides a streaming software framework similar to Apache Flink. Its core feature is a rule engine that allows users to create edge analytics applications using SQL or graph-based rules (similar to Node-RED).

  8. Overview of eKuiper Plugin Development

    eKuiper provides a plugin mechanism based on Go's plugin system to implement custom sources, sinks, and SQL functions. Due to Go plugin limitations, the following constraints apply:

    • Plugins are not supported on Windows.
    • The compilation environment must be nearly identical to the eKuiper environment, specifically requiring the same Go version, identical versions of shared dependency packages, and the same GOPATH.

    Failure to match these environments often results in plugins compiled in development failing to run in production.

  9. Overview of eKuiper use cases across industries

    eKuiper is a lightweight stream processing engine designed for real-time analysis and intelligent decision-making. It is particularly effective when deployed at the edge, close to data sources. Key industry applications include:

    1. Industrial IoT (IIoT): Used in smart manufacturing for edge-side real-time analysis and decision-making.
    2. Internet of Vehicles (IoV): Combined with the QUIC protocol to create vehicle-to-cloud system solutions that ensure easy deployment and portability for high-growth vehicle data collection.
    3. Public Data Analysis: Used to extract valuable information from various public data sharing platforms.

  10. Overview of eKuiper Wasm Plugins

    Wasm plugins complement native plugins by allowing functionality to be implemented in any language supported by WebAssembly (e.g., Go, Rust) and run in a generic environment. The general workflow for creating a plugin is: develop the plugin, build/package it according to the language, and register it via eKuiper files, REST API, or CLI.

  11. Overview of MQTT Source Connector

    The MQTT Source Connector allows eKuiper to subscribe to messages from an MQTT broker and channel them into its processing pipeline for real-time data processing. It can function as both a stream source (ingesting data) and a sink connector (publishing data).

  12. Overview of eKuiper Query Language Elements

    eKuiper uses a SQL-like language to build queries for stream processing. Key elements include:

    • SELECT: Retrieves columns from streams.
    • FROM: Specifies the required input stream.
    • JOIN: Combines multiple streams (requires a window).
    • WHERE: Filters rows based on conditions.
    • GROUP BY: Groups rows into summary rows (requires a window).
    • ORDER BY: Orders the resulting rows.
    • HAVING: Filters groups or aggregates.
    • LIMIT: Restricts the number of output records.