DOCSEARCHbyAlgolia
Submit Docs

EventFlux Documentation

website·Indexed 30 days ago

https://eventflux.io

A high-performance Complex Event Processing (CEP) engine built in Rust. EventFlux features an extended SQL syntax for stream processing, including pattern matching, windowing, and aggregations. It provides a modular architecture with a lock-free event pipeline, enterprise-grade state management with incremental checkpointing, and a connector system for RabbitMQ, WebSockets, and local files.

Tokens
43.9K
Snippets
231
Records
277
Agent score
46%

What's inside EventFlux

  1. Overview of EventFlux CEP Engine

    EventFlux is a high-performance Complex Event Processing (CEP) engine built in Rust. It is designed for latency-sensitive applications such as financial trading, IoT analytics, and real-time fraud detection, capable of processing over 1 million events per second with zero garbage collection pauses. Key technical advantages include lock-free pipelines using crossbeam queues, zero-allocation hot paths via object pooling, and Rust's memory safety.

  2. Overview of EventFlux Complex Event Processing engine

    EventFlux is a high-performance Complex Event Processing (CEP) engine built in Rust. It is designed to provide low latency and high throughput without the need for complex configurations. Key features include a SQL-first approach using SQL with streaming extensions, Rust-based performance (utilizing zero garbage collection and lock-free data structures), and enterprise-ready stateful processing with support for checkpointing and recovery.

  3. Overview of EventFlux Mappers

    Mappers in EventFlux transform data between raw bytes and structured EventFlux events. They are responsible for serialization (encoding events to bytes) and deserialization (parsing bytes into events) during connector communication. EventFlux provides three built-in mappers: JSON, CSV, and Bytes.

  4. EventFlux Connector Architecture Overview

    EventFlux connectors are Rust types that implement the Source or Sink traits. They operate exclusively on bytes; parsing and serialization are handled by separate mappers to keep transports and formats pluggable.

    Data Flow:

    • Source: external system $\rightarrow$ bytes $\rightarrow$ SourceCallback $\rightarrow$ SourceMapper $\rightarrow$ Events $\rightarrow$ pipeline
    • Sink: pipeline $\rightarrow$ Events $\rightarrow$ per event: SinkMapper $\rightarrow$ bytes $\rightarrow$ Sink::publish() $\rightarrow$ external system

  5. Understand EventFlux configuration precedence

    EventFlux uses a hierarchical configuration system where higher-precedence sources override lower ones. The order from lowest to highest priority is: Code Defaults $\rightarrow$ Configuration File (YAML) $\rightarrow$ CLI --set flags $\rightarrow$ SQL WITH clause.
    1. SQL WITH clause (Highest Priority)
2. CLI --set flags
3. Config file (YAML)
4. Code defaults (Lowest Priority)

  6. Understand Scalar vs Aggregate functions in EventFlux

    EventFlux distinguishes between two primary function types:

    1. Scalar Functions (Stateless): Operate on individual values and return a single result. They execute once per event and have no memory of previous events. They can be used in SELECT, WHERE, and HAVING clauses. Examples include UPPER(), ABS(), and ROUND().

    2. Aggregate Functions (Stateful): Accumulate values over a group of events (typically within a window) and return a summary result. They maintain internal state across multiple events and require a WINDOW or GROUP BY context. State resets when the window expires. Examples include SUM(), AVG(), COUNT(), FIRST(), and LAST().

    -- Scalar Example
SELECT symbol, UPPER(name) AS upper_name FROM Trades;

-- Aggregate Example
SELECT symbol, SUM(volume) AS total_volume 
FROM Trades 
WINDOW TUMBLING(5 min) 
GROUP BY symbol;

  7. EventFlux Connector Architecture Overview

    EventFlux uses a connector system to integrate with external messaging systems and data stores. The pipeline follows a specific flow: Source → Source Mapper (decode) → Query Engine → Sink Mapper (encode) → Sink.

    • Sources: Consume data from external systems, handling connection management, reconnection, message acknowledgment, and backpressure.
    • Sinks: Publish processed events to external systems. Each event is serialized and published as a single transport message; a failure to publish one event does not drop others in the same batch.
    • Mappers: Transform raw bytes into structured events (Source Mappers) or structured events into output formats (Sink Mappers).

  8. Understand the EventFlux SQL Compiler pipeline

    The EventFlux SQL compiler transforms queries into optimized execution plans through three main stages:

    1. Parser (sql_compiler): A fork of datafusion-sqlparser-rs extended with EventFluxDialect to support streaming constructs like WINDOW, MATCH, and WITHIN clauses.
    2. Semantic Analyzer: Performs type checking, validation, stream/table schema resolution, and expression type inference.
    3. Query Planner: Generates the logical plan, optimizes the query, and creates the final physical plan.

  9. Understand EventFlux configuration precedence

    EventFlux uses a layered configuration model where higher layers override lower layers. This allows developers to set broad defaults and refine them for specific environments or individual streams. The order of precedence (from lowest to highest) is:

    1. Code Defaults: Built-in defaults defined within the EventFlux source code.
    2. Configuration File: Settings defined in eventflux.yaml or other discovered config files.
    3. CLI Overrides: Values passed via the --set flag at runtime.
    4. SQL WITH Clause: Settings defined directly within the CREATE STREAM statement in the SQL file. These have the highest priority and cannot be overridden by other layers.

  10. Implement time-based windowing and manual time advancement

    EventFlux supports windowing (e.g., TUMBLING) within the query definition. For testing or replay scenarios, you can manually advance the system's event time using runtime.advance_time(timestamp) to trigger window evaluations.
    let app = r#"
    DEFINE STREAM Trades (
        symbol STRING, price DOUBLE, volume INT
    );
    SELECT symbol, AVG(price) AS avg_price, SUM(volume) AS total_volume
    FROM Trades
    WINDOW TUMBLING(5 sec)
    GROUP BY symbol
    INSERT INTO TradeStats;
";

let runtime = manager.create_runtime(app)?;

// Send event
runtime.send("Trades", event!["AAPL", 150.25, 100])?;

// Advance time by 5 seconds to trigger window evaluation
runtime.advance_time(1699900005)?;

  11. Load connectors as dynamic extensions

    Connectors can be compiled as separate crates into dynamic libraries and loaded at runtime using manager.set_extension(...). These libraries must export registration callbacks such as register_sources and register_sinks.

  12. Best practices for EventFlux state management

    To maintain system stability and performance, follow these state management guidelines:

    Operational Recommendations:

    • Checkpoint frequently to minimize data loss during failures.
    • Enable compression to reduce storage requirements and I/O overhead.
    • Monitor state size, as large states increase checkpoint duration.
    • Implement a cleanup strategy for old checkpoints to prevent unbounded storage growth.

    Key Considerations:

    • Large windows or complex patterns will increase the size of checkpoints.
    • Checkpointing may cause brief pauses in event processing (latency).
    • Use high-reliability storage backends for production environments.
    • Be aware that larger state sizes lead to longer recovery times during restarts.