Apache Iceberg Rust

Apache Iceberg Rust is a Rust implementation of the Apache Iceberg specification. The project is modular and composed of several specialized crates for core functionality, catalog integration, storage, and query engine integration:

Core & Loading

• iceberg: The main implementation crate.
• iceberg-catalog-loader: Logic for loading catalogs.

Catalog Implementations

• iceberg-catalog-glue: AWS Glue integration.
• iceberg-catalog-hms: Hive Metastore integration.
• iceberg-catalog-rest: REST Catalog integration.
• iceberg-catalog-s3tables: S3 Tables integration.
• iceberg-catalog-sql: SQL-based catalog integration.

Integrations & Storage

• iceberg-datafusion: Integration with the Apache DataFusion query engine.
• iceberg-cache-moka: Integration with the Moka caching library.
• iceberg-storage-opendal: Storage backend support via OpenDAL.

The iceberg-catalog-rest crate provides the official native Rust implementation for interacting with an Apache Iceberg REST Catalog. This allows Rust applications to manage and discover Iceberg tables via the REST protocol.

For detailed usage instructions and the complete API surface, refer to the official API documentation.

This crate provides the official Native Rust implementation for interacting with the Apache Iceberg Glue Catalog. It allows Rust applications to manage and discover Iceberg tables stored using AWS Glue as the catalog.

For detailed usage instructions and the full API surface, refer to the official API documentation.

The iceberg-catalog-hms crate provides the official native Rust implementation for interacting with an Apache Iceberg HiveMetaStore (HMS) Catalog. This allows Rust applications to manage and discover Iceberg tables stored using the Hive Metastore service.

For detailed API usage and code examples, refer to the official API documentation.

Apache Iceberg is an open table format for huge analytic datasets. It provides a metadata layer that sits on top of file formats like Parquet and ORC. This layer enables SQL-like table capabilities in processing engines (such as Spark, Trino, PrestoDB, Flink, Hive, and Impala) and provides features like:

• Transactional consistency
• Schema evolution
• Time travel

Iceberg is designed to work with data stored directly on object storage systems like Amazon S3.

The StorageFactory trait is used to create Storage instances from configuration. This pattern allows for lazy initialization of storage backends and enables custom storage injection into the system. A factory is used to build a new Storage instance given a StorageConfig.

#[typetag::serde(tag = "type")]
pub trait StorageFactory: Debug + Send + Sync {
    /// Build a new Storage instance from the given configuration.
    fn build(&self, config: &StorageConfig) -> Result<Arc<dyn Storage>>;
}

The following core protocol traits are hosted in the iceberg crate. These define the fundamental behavior of the Iceberg specification and are intended to be implemented by various companion crates (e.g., FileIO implementations in crates/fileio/* or Runtime implementations in crates/runtime/*):

• FileIO: Handles file system operations.
• Runtime: Manages the execution environment (e.g., iceberg-runtime-tokio).
• Catalog: Manages table namespaces and references.
• Table: Represents an Iceberg table.
• Transaction: Manages atomic updates to the table.
• TableScan: Describes a plan for scanning table data.

The storage logic is organized across several crates:

• iceberg (Core):
  • io/storage.rs: Defines Storage and StorageFactory traits.
  • io/file_io.rs: Defines FileIO, InputFile, and OutputFile.
  • io/config/: Contains StorageConfig and backend-specific constants (S3, GCS, OSS, Azdls).
  • io/memory.rs: Provides MemoryStorage (in-memory HashMap for testing).
  • io/local_fs.rs: Provides LocalFsStorage (standard filesystem operations).
• iceberg-storage-opendal (Extension):
  • Provides OpenDalStorage and OpenDalStorageFactory for cloud providers like S3, GCS, Azure, and OSS via the OpenDAL framework.
• catalog implementations (e.g., rest, glue, hms, s3tables, sql):
  • These use with_storage_factory injection to support various storage backends.