DOCSEARCHbyAlgolia
Submit Docs

Apache Fluss Documentation

repository·main·Indexed 24 days ago

https://github.com/apache/fluss

A streaming storage engine serving as a real-time data layer for Lakehouse architectures, unifying streaming and historical data for analytics and AI workloads. Includes documentation for Flink integration, data tiering to Apache Paimon and Iceberg, and client libraries for Rust, Python, Elixir, and C++.

Tokens
330.7K
Snippets
679
Records
1.1K
Agent score
82%

What's inside Apache Fluss

  1. What is Apache Fluss?

    main

    Apache Fluss is a streaming storage system designed for real-time analytics and AI. It serves as a real-time data layer for Lakehouse architectures, bridging the gap between data streaming and data Lakehouses.

    Key capabilities include:

    • Low-latency ingestion: Enables sub-second data freshness for real-time decision-making.
    • Unified Abstraction: Uses tables to unify real-time and historical data across compute engines like Apache Flink and Apache Spark (with StarRocks support coming soon).
    • Columnar Streaming: Built on Apache Arrow, supporting database primitives like column pruning and predicate pushdown to minimize I/O.
    • Compute–Storage Separation: Fluss manages state and storage (including deduplication, partial updates, delta joins, and aggregation merge engines) while stream processors handle computation.
    • ML & AI Support: Provides a unified layer for row-based, columnar, vector, and multi-modal data, suitable for real-time feature stores.
    • Observability: Built-in changelog generation provides an append-only history for auditing and reproducibility.

  2. Introduction to Apache Fluss Client Libraries

    main

    Apache Fluss is a streaming storage system designed for real-time analytics and Lakehouse architectures. The client libraries (available for Java, Rust, Python, and C++) enable developers to interact with the Fluss cluster to:

    • Create and manage databases, tables, and partitions.
    • Write data to log tables (append-only) and primary key tables (supporting upsert and delete).
    • Read data through log scanning and key lookups.
    • Integrate with the broader Fluss ecosystem, including lakehouse snapshots.

  3. Overview of Fluss Flink Tiering

    main

    The fluss-flink-tiering module provides the infrastructure required to tier Fluss data into lake formats, such as Apache Paimon. It is designed to handle the core configuration and job graph logic necessary for this process.

    It consists of two primary components:

    1. FlussLakeTiering: Encapsulates the core configuration and the Flink job graph logic.
    2. FlussLakeTieringEntrypoint: Serves as the official Flink job main class and entrypoint for executing tiering jobs.

  5. Client Feature Support Matrix

    main

    Fluss provides official clients for Java, Rust, Python, and C++. Feature support varies by client and table type (Log vs. Primary Key).

    Data Operations

    Data operations are accessed via the TableAppend, TableScan, TableUpsert, and TableLookup interfaces.

    • Log Tables: Support Append, Scan, Scan with Projection, and Typed Scan. The Java client additionally supports Typed Append and Batch Scan with Limit.
    • Primary Key Tables: Support Upsert, Upsert with Partial Update, Delete, Lookup, and Batch Scan. The Java client provides the most comprehensive support, including Typed variants and Prefix Lookup.

    Admin Operations

    Administrative tasks are managed through the FlussAdmin interface.

    • Database Management: Creating, dropping, checking existence, and listing databases (Supported by Java and Rust).
    • Table Management: Creating, dropping, altering, and listing tables (Full support across most clients).
    • Partition Management: Creating, dropping, and listing partition info (Primarily Java).
    • Cluster & Server Management: Rebalancing, altering cluster configs, and managing server tags (Primarily Java).
    • ACL Management: Creating, dropping, and listing ACLs (Java only).

  6. What is a Streaming Lakehouse with Fluss?

    main

    A Streaming Lakehouse is a unified architecture that combines real-time data streaming with long-term Lakehouse storage. Fluss achieves this by maintaining two layers of data:

    1. Real-time Data Layer (Fluss Cluster): Stores data in a streaming Arrow format optimized for sub-second low-latency reads and writes. This layer is ideal for short-term storage (e.g., retaining days of data).
    2. Historical Data Layer (Lakehouse Storage): Stores compacted data in Parquet format with high compression, optimized for efficient analytics and long-term storage (e.g., retaining months of data).

    Fluss unifies these layers through shared data and metadata, providing several key benefits:

    • Unified Metadata: Manage a single table to access real-time, historical, or combined data.
    • Union Reads: Compute engines (like Flink and Spark) can read the union of real-time streaming data and historical Lakehouse data seamlessly.
    • Real-Time Lakehouse: Enables truly real-time analytics by bridging the gap between streaming and batch data.
    • Analytical Streams: Provides streaming applications with powerful analytical capabilities.
    • Ecosystem Connectivity: Keeps table metadata in sync with data lake catalogs, allowing external engines like Spark, StarRocks, Flink, and Trino to read data directly via the catalog.

  7. Overview of Authentication in Fluss

    main

    Fluss uses a pluggable authentication mechanism managed through listeners. Each connection triggers a specific protocol based on the listener's configuration.

    Supported mechanisms include:

    • PLAINTEXT: The default mode. No identity verification is performed. Suitable for local development, debugging, or trusted internal clusters.
    • SASL: Based on the Simple Authentication and Security Layer. Currently, only SASL/PLAIN (username/password) is supported. Recommended for production.
    • Custom plugins: Developers can extend authentication via specific plugin interfaces.

    You can map different protocols to specific listeners using the security.protocol.map property in conf/server.yaml.

  8. What is the Aggregation Merge Engine?

    main

    The Aggregation Merge Engine is used when you only need aggregated results rather than individual records. It processes incoming data by applying a specified aggregate function to each value field associated with a primary key.

    When new data arrives with an existing primary key, the engine updates the existing record by aggregating the new values with the current ones. If no aggregate function is specified for a field, it defaults to last_value_ignore_nulls behavior.

    Common use cases include:

    • Computing running totals and statistics
    • Maintaining counters and metrics
    • Tracking maximum/minimum values over time
    • Building real-time dashboards and analytics

  9. Overview of the Tiering Service

    main

    The Tiering Service is an Apache Flink-based job that continuously compacts real-time data from Fluss into a configured data lake format (Paimon, Iceberg, Hudi, or Lance) for cost-efficient long-term storage and analytics.

    It provides:

    • Incremental Syncing: Only missing data segments are synced.
    • Exactly-once Semantics: Maintains consistency between Fluss and the data lake.
    • Automated Lifecycle: Operates by reading from Fluss tables with lakehouse storage enabled and writing to the target lake format.

  10. Use column pruning in Log Table queries

    main

    Log Tables are stored in Apache Arrow (columnar) format by default. This allows Fluss to perform column pruning during streaming reads. When a query engine (like Flink) requests only specific columns, Fluss skips reading the irrelevant columns from storage, reducing I/O and network costs.

    Example query that prunes address and amount:

    SELECT order_id, item_id FROM log_table WHERE dt > '2023-01-01';

  11. Use Pushdown Aggregations for COUNT(*)

    main

    Fluss supports pushdown COUNT(*) aggregation in batch mode for both Log Tables and Primary Key Tables. This allows for efficient row counting without a full table scan.

    Requirement for Primary Key Tables

    For Primary Key Tables, COUNT(*) pushdown requires the table to use the default changelog mode: 'table.changelog.image' = 'FULL'. Tables configured with 'table.changelog.image' = 'WAL' do not support this feature.

    -- Example for Log Table or Primary Key Table (if using FULL changelog image)
SET 'execution.runtime-mode' = 'batch';
SET 'sql-client.execution.result-mode' = 'tableau';

SELECT COUNT(*) FROM log_table;

  12. How Fluss enables stateless streaming ETL

    main

    Fluss allows stateful streaming ETL pipelines (joins, aggregations, deduplication, etc.) to move state out of the processing engine (like Flink) and into PrimaryKey Tables. This makes the processing jobs stateless, simplifying scaling and recovery.

    Common patterns include:

    • Delta Joins: Dual-stream joins are replaced by stateless index-key lookups.
    • Rolling Aggregations: Rolling counts and velocity signals are handled via writes to the Aggregation Merge Engine.
    • Wide-row Assembly: Multi-producer updates are handled via partial-updates against a shared row.