DOCSEARCHbyAlgolia
Submit Docs

Apache Fluss Documentation

website·Indexed 20 days ago

https://fluss.apache.org/

Documentation for Apache Fluss, a streaming lakehouse that provides unified stream and lakehouse capabilities. It includes guides on table design (Primary Key, Log, and Partitioned tables), integration with Apache Flink and Apache Spark, and support for LakeHouse storage via Apache Iceberg and Paimon. The docs cover API usage for C++ and Java, cluster maintenance, tiered storage, and security configuration including authentication and ACLs.

Tokens
442.7K
Snippets
2.1K
Records
3K
Agent score
95%

What's inside Apache Fluss

  1. Overview of Apache Fluss streaming storage

    Apache Fluss is a streaming storage system built for real-time analytics, serving as a real-time data layer for Lakehouse architectures. It bridges streaming data and data Lakehouses by providing low-latency, high-throughput data ingestion and processing.

    Key technical capabilities include:

    • Performance: Sub-second latency for streaming reads and writes.
    • Storage: Data is stored in a columnar format to improve query performance and reduce costs.
    • Table Types: Supports append-only 'Log Tables' and updatable 'PrimaryKey Tables'.
    • Integrations: Seamlessly integrates with Apache Flink (with Apache Spark and StarRocks support planned).
    • Reliability & Scale: Features built-in replication for fault tolerance, horizontal scalability, high-QPS lookup joins, and bulk read/write operations.

  2. Overview of the Fluss Tiering Service

    The Fluss tiering service is designed to manage the lifecycle of data by moving it from high-performance local disks (the 'hot' layer) to a Data Lakehouse (the 'cold' layer) such as S3 or GCS using open table formats like Iceberg, Paimon, Hudi, or Lance. This process ensures that Fluss maintains a manageable working set of data while providing long-term storage for analysts and batch jobs. The service achieves three primary goals: keeping the Lakehouse fresh (within a configurable target), allowing Fluss to reclaim local disk space after data is safely tiered, and ensuring that updates to the Lakehouse are atomic.

  3. Overview of Fluss streaming storage

    Fluss is a streaming storage system designed for real-time analytics and AI, serving as a real-time data layer for Lakehouse architectures. It bridges streaming data and data Lakehouses by providing low-latency, high-throughput ingestion and processing. It stores data in a columnar format to optimize query performance and reduce storage costs, supporting sub-second latency for streaming reads and writes. Key features include built-in replication for fault tolerance, horizontal scalability, high-QPS lookup joins, and bulk read/write operations. It currently integrates with Apache Flink, with support for Apache Spark and StarRocks planned.

  4. Overview of the Apache Fluss Three-Tier Storage Hierarchy

    Apache Fluss utilizes a three-tier storage model to balance latency, cost, and analytical accessibility. Each record typically exists in only one tier at a time:

    1. Tier 1: Local Disk (Tablet Server): Stores 'hot' data, including recent log segments and the live RocksDB state for primary-key tables. Provides millisecond-level read latency.
    2. Tier 2: Remote Object Storage (S3, GCS, etc.): Used for two purposes: extending log retention via the remote-log tiering task (native binary format) and storing durable KV snapshots for primary-key table recovery. Configured via remote.data.dir.
    3. Tier 3: Lakehouse (Paimon, Iceberg, Lance): Stores data in analytical file formats for query engines like Spark, Trino, and Flink. Reads from this tier typically take seconds.

  5. Overview of Fluss Security Model

    Fluss employs a pluggable security model consisting of two core components: Authentication (verifying client identity) and Authorization (controlling resource access and operations). By default, neither is enabled, allowing all clients unrestricted access, which is suitable for development but not for production.

  6. Overview of the Aggregation Merge Engine

    The Aggregation Merge Engine is used when only aggregated results are needed instead of individual records. It aggregates value fields for the same primary key using a specified aggregate function. If no function is specified for a field, it defaults to last_value_ignore_nulls. This engine is ideal for real-time counters, running totals, and analytics dashboards.

  7. Overview of Fluss Unified Streaming & Lakehouse Architecture

    Fluss is a streaming storage system designed for real-time analytics that unifies streaming and Lakehouse architectures. It provides millisecond-level end-to-end latency for real-time reads/writes while utilizing a Tiering Service to offload older data into standard Lakehouse formats such as Apache Paimon or Apache Iceberg. This allows the system to maintain the latest high-fidelity data in Fluss and historical data in Paimon, enabling 'Union Reads' via Apache Flink to achieve second-level data freshness across the entire dataset.

  8. Overview of Fluss authentication mechanisms

    Fluss uses a pluggable authentication mechanism where each connection triggers a specific protocol based on the listener configuration. The security.protocol.map property in conf/server.yaml is used to map listeners to protocols. Supported mechanisms include:

    • PLAINTEXT: Default mode with no identity verification; suitable for local development or trusted internal clusters.
    • SASL: Based on Simple Authentication and Security Layer. Currently, only SASL/PLAIN (username/password) is supported; recommended for production.
    • Custom plugins: Extendable via interfaces for third-party or enterprise integrations.

  9. Overview of Delta Join in Apache Fluss and Flink

    Introduced in Apache Flink 2.1 and supported in Apache Fluss 0.8, the Delta Join operator optimizes streaming joins by replacing traditional state-heavy joins with an index-key lookup mechanism. Instead of storing both input sides entirely in Flink state, Delta Join queries data directly from the Fluss source table. This significantly reduces state size, lowers resource consumption, shortens checkpointing and recovery times, and reduces state bootstrap overhead.

  10. Overview of Fluss Tiered Storage Architecture

    Fluss implements a tiered storage architecture to decouple compute and storage resources, improve client isolation, and accelerate maintenance. Data is organized into different layers based on access patterns and cost:

    1. Local Storage: Stores recent data to ensure high write and read performance.
    2. Remote Storage: Stores historical data to reduce costs.
    3. Lakehouse Storage: Converts data from Fluss's native real-time format into open data lake formats to optimize batch analytics performance. Currently supported formats include Paimon, Iceberg, and Lance.

  11. Overview of Fluss Security Model

    Fluss employs a security model based on two core components: Authentication (verifying client identity) and Authorization (controlling resource access and operations). By default, neither is enabled, allowing all clients unrestricted access. Both mechanisms are pluggable to allow for custom security policies.

  12. Overview of Fluss Security Model

    Fluss employs a security model based on two core components: Authentication (verifying client identity) and Authorization (controlling resource access and operations). By default, neither is enabled, allowing all clients unrestricted access. Fluss supports pluggable mechanisms for both, allowing for custom security policies.