DOCSEARCHbyAlgolia
Submit Docs

Apache Iceberg

repository·main·Indexed 11 days ago

https://github.com/apache/iceberg

A high-performance table format for huge analytic datasets that provides SQL-like reliability and multi-engine compatibility with Spark, Flink, Trino, Presto, Hive, and Impala. Includes a core Java library, a REST Catalog specification, and various integration modules such as iceberg-spark and iceberg-flink.

Tokens
201.6K
Snippets
431
Records
730
Agent score
93%

What's inside Apache Iceberg

  2. What is Apache Iceberg?

    main
    Apache Iceberg is an open table format designed for huge analytic datasets. It allows compute engines like Spark, Trino, PrestoDB, Flink, Hive, and Impala to interact with massive datasets using a high-performance format that behaves like a standard SQL table. It is designed to provide reliability and performance on cloud object stores by solving correctness problems and enabling fast scan planning without requiring a distributed SQL engine for basic file discovery.

  3. Overview of Apache Iceberg

    main
    Apache Iceberg is a high-performance table format for huge analytic tables. It provides the reliability and simplicity of SQL tables for big data environments, enabling multiple engines (such as Spark, Trino, Flink, Presto, Hive, and Impala) to safely interact with the same tables simultaneously. The core Java library serves as the reference implementation for other language bindings.

  5. Overview of the Apache Iceberg Sink Connector

    main

    The Apache Iceberg Sink Connector is a Kafka Connect sink connector designed to write data from Kafka topics into Apache Iceberg tables. It provides several key features for reliable and scalable data ingestion:

    • Commit coordination: Centralized Iceberg commits to ensure consistency.
    • Exactly-once delivery semantics: Leverages KIP-447 for reliable data transfer.
    • Multi-table fan-out: Ability to write to multiple tables.
    • Automatic table management: Handles automatic table creation and schema evolution.
    • Field name mapping: Uses Iceberg’s column mapping functionality to map Kafka fields to table columns.

  6. New features in Apache Iceberg C++ 0.2.0

    main

    The 0.2.0 release of Apache Iceberg C++ introduces significant capabilities for table management, data access, and catalog interaction. Key improvements include:

    • Table Operations: Support for schema evolution (add, delete, update, and move columns), table property updates, sort order management, partition spec updates, and location/statistics management. It also includes a Transaction API for snapshot management (e.g., fast append).
    • Data Access: Support for v2 deletes and metadata column reads. The ManifestReader now supports projection and filtering, and a file scan task reader is available with Arrow C Stream integration.
    • REST Catalog: A full REST Catalog client is available, supporting namespace operations and table CRUD (create, load, drop, list, update, and stage-create).
    • Expression System: A complete framework including literal expressions, type casting, binary serialization, and evaluators (inclusive/strict metrics, manifest, residual, aggregate, and projection).
    • Performance: Optimized Avro reader/writer with direct encoding and multi-block support, alongside configurable Avro and Parquet readers/writers.
    • Catalog & Metadata: An InMemoryCatalog implementation for table management, along with location providers, partition path generation, and table metadata builders.

  7. Understand Iceberg Library Modules

    main

    Iceberg is organized into several modules depending on your use case:

    Core API & Implementation:

    • iceberg-api: Public API (expressions, types, tables).
    • iceberg-core: Main implementation (Avro support). Processing engines should depend on this.
    • iceberg-common: Utility classes.
    • iceberg-data: Client library for reading tables from JVM applications.

    Storage Formats & Cloud:

    • iceberg-parquet: Parquet file support.
    • iceberg-orc: ORC file support (experimental).
    • iceberg-aws: AWS S3 and AWS Glue integration.
    • iceberg-hive-metastore: Hive Metastore integration.

    Engine Integrations:

    • iceberg-spark: Spark Datasource V2 implementation.
    • iceberg-flink: Flink Table and DataStream API implementation.
    • iceberg-mr: MapReduce and Hive InputFormats.
    • iceberg-nessie: Project Nessie integration.

  8. Feature support for Apache Iceberg with Apache Flink

    main

    Apache Iceberg supports both Apache Flink's DataStream API and Table API. The following table summarizes the feature support for Flink integration:

    FeatureSupport
    SQL create catalog✔️
    SQL create database✔️
    SQL create table✔️
    SQL create table like✔️
    SQL alter table✔️ (Only supports altering table properties; column and partition changes are not supported)
    SQL drop_table✔️
    SQL select✔️ (Supports both streaming and batch mode)
    SQL insert into✔️ (Supports both streaming and batch mode)
    SQL insert overwrite✔️
    DataStream read✔️
    DataStream append✔️
    DataStream overwrite✔️
    Metadata tables✔️
    Rewrite files action✔️

  9. Overview of Iceberg Expressions

    main

    Iceberg expressions provide a common, simple structure for representing and exchanging logical expressions. They are used for:

    • Data Validations: Defining CHECK constraints.
    • Default Values: Specifying values like current_timestamp().
    • Server-side Scan Planning: Exchanging filter logic within the catalog protocol to optimize data access.

    To maintain simplicity and portability across different engines, the specification is deliberately constrained to two main categories: Value expressions and Predicates. Complex logic is handled by calling specific functions (such as partition transforms) or User-Defined Functions (UDFs) rather than embedding engine-specific SQL dialects directly into the expression structure.

  10. What is a Catalog and how does it work?

    main

    In Iceberg, a Catalog is responsible for managing a collection of tables, typically grouped into namespaces. While Iceberg manages the data format for individual tables, the catalog handles high-level operations such as:

    • Creating, dropping, and renaming tables.
    • Tracking a table's current metadata, which is provided to compute engines when a table is loaded.

    To use Iceberg, you must first initialize and configure a catalog. This is usually done through a processing engine (like Spark, Flink, or Trino) by passing a set of properties. Multiple compute engines can share a single Iceberg catalog to maintain a common data layer.

  11. Overview of Iceberg UDF and UDTF types

    main

    Iceberg supports two primary types of SQL user-defined functions (UDFs) through a standardized metadata format:

    • Scalar function (UDF): Returns a single value. This value can be a primitive type (e.g., int, string) or a non-primitive type (e.g., struct, list).
    • Table function (UDTF): Returns a table containing zero or more rows, where all rows follow a uniform schema.

  12. Catalogs and Integrations in Iceberg C++ 0.3.0

    main

    The 0.3.0 release expanded catalog and storage support:

    • REST Catalog: Added OAuth2 support (including auto-refresh), basic authentication, snapshot loading mode, namespace separators, and server-side scan planning endpoints.
    • S3 Integration: S3 FileIO integration built on Arrow filesystem support.
    • FileIO Interface: New InputFile and OutputFile interfaces, and bulk delete support.
    • SQL Catalogs: Support for catalogs backed by SQLite, PostgreSQL, and MySQL.