DOCSEARCHbyAlgolia
Submit Docs

SpringQL Documentation

website·Indexed 29 days ago

https://springql.github.io/

Documentation for SpringQL, a stream processing engine that uses SQL for defining dataflows. It features support for window aggregations, window joins, event time and watermarks, and a pipeline model consisting of source readers, pumps, and sink writers. Available versions include 0.16, 0.17, and Next.

Tokens
48K
Snippets
267
Records
359
Agent score
64%

What's inside SpringQL

  1. Overview of SpringQL Sink Writers

    A Sink Writer is responsible for moving data from a sink stream to a foreign sink. Except for the IN_MEMORY_QUEUE type, a sink writer typically performs four steps: maintaining a connection to the foreign sink, fetching rows from the sink stream, converting those rows into a supported data format (currently 1-line JSON objects), and writing the data to the destination.

  2. Overview of SpringQL Stream Processor

    SpringQL is a light-weight stream processor designed for edge computing. It runs on a single machine and is specifically intended for developing applications that process unbound data streams originating from IoT devices.

  3. Overview of SpringQL for Edge Computing

    SpringQL is a lightweight stream processor designed to run on a single machine. It is specifically built for edge computing scenarios, enabling developers to create applications that process unbound data streams originating from IoT devices.

  4. Overview of SpringQL Stream Processor

    SpringQL is a light-weight stream processor designed for edge computing that runs on a single machine. It is specifically built to help developers create applications that process unbound data streams originating from IoT devices.

  5. Overview of SpringQL Stream Processor

    SpringQL is a light-weight stream processor designed for edge computing that runs on a single machine. It is specifically built to help developers create applications that process unbound data streams originating from IoT devices.

  6. Overview of SpringQL for Edge Computing

    SpringQL is a lightweight stream processor designed to run on a single machine. It is specifically built for edge computing, enabling developers to create applications that process unbound data streams originating from IoT devices.

  7. Use windows for stream aggregation

    Because stream data is unbounded, operations like sorting, aggregation, and joins require windows. Windows split an unbounded sequence of rows into bounded sets, allowing set-based operations to be applied within the window boundaries.

  8. Understand the Continuous Query model in SpringQL

    SpringQL does not support standalone DML (INSERT, UPDATE, DELETE) or standalone SELECT queries. Instead, it uses a Continuous Query (CQL) model via CREATE PUMP statements.

    In a CREATE PUMP statement:

    • A SELECT statement is used to fetch and process rows from upstreams.
    • An INSERT statement is used to push result rows to downstreams.

    Unlike traditional relational databases where a query is executed once, SpringQL queries are registered to run continuously, applying the logic to every input row as it arrives.

  9. Understand Sink Writer functionality in SpringQL

    A Sink Writer acts as the bridge between a SpringQL sink stream and a foreign destination. Except for the IN_MEMORY_QUEUE type, a sink writer performs four primary steps: maintaining a connection (network or file handle) to the foreign sink, fetching rows from the sink stream, converting those rows into a compatible data format, and writing the data to the destination.

  10. Use Source and Sink Streams for External Data

    SpringQL distinguishes between three types of streams to handle data movement:

    • Normal Streams: Standard internal data flow.
    • Source Streams: Used to input foreign data via source readers.
    • Sink Streams: Used to output data to foreign systems via sink writers.

    Note: While defined differently in SpringQL, there are currently no functional differences between these three stream types.

  11. Understand SpringQL's system architecture

    SpringQL processes SQL commands through a multi-stage flow:

    1. A host application calls the SpringQL API with a SQL string.
    2. The SQL processor parses the SQL and converts it into internal commands.
    3. The SQL executor in the stream engine receives these commands and updates the pipeline.
    4. The autonomous executor's task executor updates the corresponding task graph based on these pipeline changes.

  12. Understand the Continuous Query model in SpringQL

    SpringQL uses a Continuous Query (CQL) model rather than the one-time execution model found in traditional relational databases. While standalone SELECT, INSERT, UPDATE, and DELETE statements are not supported, these operations are used within CREATE PUMP statements.

    In a CREATE PUMP statement:

    • A SELECT statement is used to fetch and process rows from upstreams.
    • An INSERT statement is used to push result rows to downstreams.

    Once registered, these queries run continuously and are applied to every input row that enters the system.