DOCSEARCHbyAlgolia
Submit Docs

ksqlDB

repository·master·Indexed 43 days ago

https://github.com/confluentinc/ksql

A distributed, scalable, real-time database for building stream processing applications on top of Apache Kafka using SQL syntax. This repository includes documentation on the KSQL Improvement Proposal (KLIP) process, local build instructions for documentation and Docker images, the ksql-datagen test data tool, JMH microbenchmarks, and the KSQL Query Validation Test suite.

Tokens
231.3K
Snippets
690
Records
981
Agent score
50%

What's inside ksqlDB

  2. Overview of the ksqlDB Schema Migrations Tool

    master
    The ksqlDB Schema Migrations Tool is a utility designed to provide automated schema management for ksqlDB environments. It allows developers to version control their ksqlDB schemas, recreate schemas from scratch, and migrate existing schemas to newer versions. This tool is specifically designed to integrate with CI/CD pipelines to automate schema evolution across different environments (e.g., Dev, QA, Prod).

  3. ksqlDB API Reference Index

    master

    The ksqlDB API reference provides a comprehensive guide to programming streaming applications. The API is organized into several functional areas:

    SQL Syntax and Logic

    • SQL Quick Reference: Core syntax and usage.
    • Functions and Operators: Includes Aggregate Functions, Scalar Functions, Table Functions, Operators, and Type Coercion rules.

    Data Abstractions

    • Streams and Tables: Commands for managing stream and table lifecycles, including CREATE STREAM, CREATE TABLE, CREATE STREAM AS SELECT (CSAS), CREATE TABLE AS SELECT (CTAS), DROP, DESCRIBE, and data insertion via INSERT INTO or INSERT VALUES.

    Query Execution

    • Queries: Managing query lifecycles with SELECT (supporting both Pull and Push queries), EXPLAIN for query plans, and control commands like PAUSE, RESUME, and TERMINATE.

    Integration and Customization

    • Connectors: Managing Kafka Connectors via CREATE CONNECTOR, DESCRIBE CONNECTOR, DROP CONNECTOR, and SHOW CONNECTORS.
    • Custom Types: Defining and managing user-defined types with CREATE TYPE, DROP TYPE, and SHOW TYPES.

    System Management and Metadata

    • Metadata: Inspecting the system state using SHOW commands (Streams, Tables, Topics, Functions, Properties, Queries) and DESCRIBE commands.
    • Execution: Running automation via RUN SCRIPT and managing system settings with ALTER SYSTEM.
    • Assertions: Validating system state with ASSERT SCHEMA and ASSERT TOPIC.

  4. Understand ksqlDB Architecture and Components

    master

    ksqlDB is a database purpose-built for stream processing that allows you to build event streaming applications using only SQL statements. It is built on top of Kafka Streams.

    Key components include:

    • ksqlDB Engine: Processes SQL statements and queries by parsing them and building corresponding Kafka Streams topologies.
    • REST Interface: Enables client access to the engine from the CLI, UIs, or any other REST client.
    • ksqlDB CLI: A console providing a command-line interface for interacting with ksqlDB Server instances.
    • ksqlDB UI: A graphical interface for developing applications (available in Confluent Cloud and Confluent Control Center).

    ksqlDB Server comprises both the ksqlDB engine and the REST API. Multiple server instances can be added to a cluster to scale load without restarting applications.

  5. Supported operations in the ksqlDB Java client

    master

    The ksqlDB Java client is designed to support a specific subset of ksqlDB operations. When building applications using the Java client, you can expect support for the following categories of operations:

    Query Operations

    • Push queries: Continuous streaming queries.
    • Pull queries: Request-response style queries.

    DDL (Data Definition Language) Operations

    • CREATE STREAM
    • CREATE TABLE
    • CREATE STREAM ... AS SELECT
    • CREATE TABLE ... AS SELECT
    • DROP STREAM
    • DROP TABLE
    • INSERT INTO
    • TERMINATE <queryId>

    Admin Operations

    • SHOW TOPICS (non-extended)
    • SHOW STREAMS (non-extended)
    • SHOW TABLES (non-extended)
    • SHOW QUERIES (non-extended)

    Data Manipulation and Management

    • Inserting values (inserting rows into an existing stream).
    • Terminating push queries via the /close-query endpoint.

  6. Use ksqlDB functions for stream processing

    master

    ksqlDB provides three primary categories of functions to transform and analyze data in streaming applications:

    1. Aggregate Functions: Used to perform computations over groups of rows (e.g., calculating sums, averages, or counts over a time window).
    2. Scalar Functions: Used to transform individual rows or values (e.g., string manipulation, mathematical operations, or date/time conversions).
    3. Table Functions: Used to perform lookups or join streaming data against external or static data sources.

    Refer to the specific documentation for each category to see the full list of available functions and their signatures.

  7. Available ksqlDB clients

    master

    ksqlDB supports several language-specific clients to send requests to the server. There is one official client and several community-maintained unofficial clients.

    Official Client

    • Java client: The primary official client for ksqlDB applications.

    Unofficial (Community) Clients

    • .NET client: Maintained by community members.
    • Go client: Maintained by community members.
    • Python client: Maintained by community members.

    Note: ksqlDB is no longer accepting community contributions for new client libraries.

  8. Explore ksqlDB Reference Material

    master

    The ksqlDB reference documentation provides technical details across several key areas required to build and operate stream processing applications:

    Language and Logic

    • The SQL language: Learn the lexical structure and syntax required to write ksqlDB programs.
    • Built-in functions: Documentation for the functions shipped with ksqlDB that can be invoked in queries.
    • Operators: Details on the built-in operators available in the language.
    • Appendix: A quick reference guide for the various elements of the ksqlDB language.

    Server and Operations

    • Server configuration: Instructions on how to control ksqlDB server behavior via configuration.
    • Metrics: Information on how to observe and monitor cluster behavior over time.
    • Processing log: Details on logging internal events that occur during row processing.

    Extensibility and Data Handling

    • User-defined functions (UDFs): Guidance on extending ksqlDB by writing and invoking custom Java code.
    • Serialization: How to control the format of data as it is read from or written to streams.

    Client Integration

    • Clients: Information on interacting with ksqlDB using various programming languages.
    • REST API: Documentation for communicating with ksqlDB via an HTTP REST API.

  10. Core ksqlDB Concepts

    master

    ksqlDB is built around several fundamental abstractions for stream processing. Understanding these concepts is essential for building streaming applications:

    Data Units and Collections

    • Events: The fundamental unit of data in stream processing.
    • Streams: An immutable, append-only collection of events representing a series of historical facts.
    • Tables: A mutable collection of events that models change over time (representing the current state).

    Processing and Retrieval

    • Stream processing: The method of writing programs that compute over unbounded streams of events.
    • Queries: The mechanism used to process events and retrieve computed results.
    • Materialized views: Precomputed results of queries stored at write-time to ensure predictably fast reads.
    • Joins: The process of combining data from multiple streams and tables into a single result.

    Extensions and Integration

    • User-defined functions (UDFs): Allows extending ksqlDB by invoking custom code written in Java.
    • Lambda Functions: Enables applying in-line functions without the overhead of creating a full UDF.
    • Connectors: Components used to source data from or sink data to external systems.
    • Windows: Used to bound continuous streams of events into distinct, manageable time intervals.

  11. Overview of YATT (Yet Another Testing Tool)

    master

    YATT (Yet Another Testing Tool) is a proposed SQL-driven testing tool for ksqlDB designed to provide coverage for complex test cases that require interleaving statements, inserts, and assertions.

    Unlike existing tools like ksql-test-runner or (R)QTT, YATT is intended to be imperative and written primarily in SQL. This allows developers to stay within a "SQL mindset" and use standard SQL expressions to produce test data, rather than relying on custom JSON formats or heavy Java-based testing frameworks.

    Key design principles include:

    • Support for imperative tests (interleaved statements/inserts/asserts).
    • Primary use of SQL syntax to minimize special-syntax directives.
    • Leveraging the ksqlDB engine directly to minimize testing-specific code.
    • Support for fast, parallelized test execution.

  12. What is the ksqlDB Processing Log?

    master

    The processing log is a stream of record processing events emitted by ksqlDB to help debug SQL queries. As ksqlDB executes a query, it writes structured events to this log detailing how each row is processed and any errors encountered.

    Unlike standard server logs, the processing log is designed for per-record debugging and can be consumed by downstream applications or by ksqlDB itself as a ksqlDB stream.

    Important Distinction: The processing log is NOT for server logging. To configure server logs (e.g., using a kafka_appender), you must configure the io.confluent.ksql logger in the ksqlDB Server config, not the processing logger.