DOCSEARCHbyAlgolia
Submit Docs

iceberg-rust

repository·main·Indexed 42 days ago

https://github.com/jankaul/iceberg-rust

A Rust implementation of the Apache Iceberg specification designed for interoperability with the Arrow ecosystem. It provides ACID guarantees for large analytic datasets and includes deep integration with the DataFusion query engine via the datafusion_iceberg crate (v0.10.0). Supports various catalogs including REST, S3Tables, Filesystem, Glue, and RDBMS (Postgres, MySQL), as well as Parquet file format.

Tokens
52.4K
Snippets
145
Records
216
Agent score
53%

What's inside iceberg-rust

  1. Overview of Iceberg-rust features

    main

    Iceberg-rust is a low-level Rust implementation of the Apache Iceberg specification. It provides the following core abstractions for working with Iceberg tables:

    • Catalog Trait: A unified interface to interact with various Iceberg catalog implementations.
    • Table Structs: High-level structures that simplify accessing and interacting with Iceberg table metadata.
    • Table Transactions: Mechanisms to perform atomic changes to Iceberg table metadata, ensuring consistency.
    • View Structs: Data structures used to define and manage Iceberg views.

  3. Use Apache Iceberg with DataFusion

    main

    The datafusion_iceberg crate provides integration between Apache Iceberg and DataFusion. It implements the core DataFusion traits required to treat Iceberg tables as DataFusion tables, specifically:

    • TableProvider: Allows DataFusion to read from and write to Iceberg tables.
    • SchemaProvider: Enables DataFusion to resolve schemas for Iceberg tables.
    • CatalogProvider: Allows DataFusion to discover and manage Iceberg catalogs.

  4. Schema Projection and Selection

    main

    You can derive new schemas from existing ones using projection.

    • select_not: Drops specified fields. If a Struct is left with no fields after selection, the struct itself is removed from the schema.
    • project: Returns a new schema containing only the specified fields. This operation preserves the underlying payloads of complex types like PrimitiveType::Variant or Geometry.

  5. Schema Compatibility Rules

    main

    When performing write operations, the system checks for compatibility between the read schema and the write schema. Key compatibility constraints include:

    • Nullability: Writing an optional field into a required schema field is a violation.
    • Type Promotion: Primitive types must follow the allowed promotion matrix.
    • Structural Integrity: Mismatches between Struct and Primitive shapes at the same field ID are rejected.
    • Collections:
      • For Map, both key and value types must be compatible, and the value_required flag must be respected.
      • For List, the element type must be compatible, and the element_required flag must be respected.
    • Field Ordering: By default, write-side reordering of fields in a Struct is rejected. This can be bypassed if check_ordering=false is used.
    • Variant Type: Any non-Variant type being written to a Variant field is rejected with an error: "cannot be read as a variant".

  6. Schema Field ID and Name Lookup

    main
    Iceberg schemas support looking up fields by their unique integer ID or their string name. This is critical for maintaining data integrity across schema evolutions. The system supports nested path resolution (e.g., a.b.c) and case-insensitive selection via Schema::case_insensitive_select.

  9. Overview of the iceberg-rust-spec crate

    main

    The iceberg-rust-spec crate is the core implementation of the Apache Iceberg table format specification in Rust. It provides the fundamental building blocks for interacting with Iceberg tables, including metadata serialization, schema management, and Arrow integration.

    Key capabilities include:

    • Core Specification: Complete implementation of the Apache Iceberg table format.
    • Metadata Management: Type-safe representations of schemas, partitioning, sorting, and other metadata, including full serialization/deserialization support.
    • Arrow Integration: Support for reading and writing Iceberg data using Apache Arrow.
    • Utilities: Common helper functions for table operations.

  10. Core components of iceberg-rust

    main

    The iceberg-rust crate is organized into several functional modules that handle different aspects of the Iceberg specification:

    • table: Core table operations and management.
    • catalog: Implementations for metadata storage (e.g., REST, AWS Glue, File-based).
    • arrow: Integration with Apache Arrow for data processing.
    • view: Support for Iceberg Views.
    • materialized_view: Support for Iceberg Materialized Views.
    • spec: Core Iceberg specification types and logic.
    • file_format: Handling of underlying data file formats.
    • object_store: Abstractions for interacting with storage (e.g., S3).
    • sql: SQL-related functionality.
    • config: Configuration management.
    • error: Error types and error handling.

  11. How FirstRowIdInheritance works for manifest entries

    main

    FirstRowIdInheritance is used to manage the propagation of first_row_id values across manifest entries. This is particularly relevant when dealing with committed manifests to ensure lineage is preserved or correctly assigned.

    Key behaviors:

    • Inheritance: It can assign first_row_id values to entries that currently have None based on the order of entries in the manifest.
    • Committed vs Uncommitted:
      • For committed manifests, it can clear existing first_row_id values that were written before an upgrade.
      • For uncommitted manifests, it preserves explicitly set first_row_id values on entries while applying inheritance to others.
    • Error Handling: It will return an Error::InvalidFormat if a counter overflow occurs during the application process.
    // Example of applying inheritance to a list of entries
let mut inheritance = FirstRowIdInheritance::for_committed_manifest(Some(10));
for entry in &mut entries {
    inheritance.apply(entry).unwrap();
}

  12. Value casting rules and restrictions

    main

    The Value::cast method allows converting one Value to another type. However, many casts are restricted based on the target PrimitiveType.

    Valid Casts (Implemented/Planned)

    • Int $\rightarrow$ Long/Date: Value::Int can be cast to Long or Date.
    • Long $\rightarrow$ Time/Timestamp/Timestamptz: Value::LongInt can be cast to these temporal types.
    • Binary $\rightarrow$ Fixed: Succeeds if bytes.len() == n (where n is the fixed size).
    • Fixed $\rightarrow$ Binary: Yields Binary(bytes) without size checks.
    • String $\rightarrow$ Fixed: Decodes a hex string and performs a length check.
    • String $\rightarrow$ Binary: Decodes a hex string into bytes.

    Invalid Casts (Will reject/error)

    • Boolean: Rejects all non-boolean target types.
    • Float/Double: Rejects all non-float target types.
    • Date/Time/Timestamp: Rejects all non-matching temporal target types.
    • Decimal: Rejects target types that are not Decimal (though different precision/scale variants are allowed).
    • UUID/String/Fixed/Binary: Rejects all non-matching target types.