icechunk

Icechunk is an open-source (Apache 2.0) transactional storage engine for tensor/ND-array data, specifically designed for use on cloud object storage. It augments the Zarr core data model with features that enhance performance, collaboration, and safety in cloud computing environments.

Key capabilities include:

• Object storage optimization: Designed for modern cloud object storage without requiring an external database.
• Serializable isolation: Provides consistent, isolated reads and atomic writes, allowing multiple uncoordinated processes to read and write safely.
• Time travel: Previous snapshots remain accessible after new ones are written.
• Data version control: Supports both branches (mutable references) and tags (immutable references).
• Chunk sharding: Decouples chunk storage from specific file names, allowing multiple chunks to be packed into a single object.
• Chunk references: Allows referencing Zarr-compatible chunks within other file formats like HDF5 or NetCDF as "virtual datasets."
• Schema evolution: Supports adding, renaming, or removing arrays and groups with minimal overhead.

Icechunk is designed around the following core capabilities:

• Object storage: Designed for modern cloud object storage (though it also works with file storage). No external database or catalog is required to maintain a repo.
• Serializable isolation: Reads are isolated from concurrent writes and always use a committed snapshot. Writes are committed atomically and are never partially visible. No locks are required for reading.
• Time travel: Previous snapshots of a repo remain accessible even after new snapshots are written.
• Data version control: Supports tags (immutable references to snapshots) and branches (mutable references to snapshots).
• Chunk shardings: Decouples chunk storage from specific file names, allowing multiple chunks to be packed into a single object.
• Chunk references: Allows Zarr-compatible chunks within other file formats (e.g., HDF5, NetCDF) to be referenced.
• Schema evolution: Supports adding, renaming, and removing Arrays and Groups within the hierarchy with minimal overhead.

An Icechunk repository is a storage specification for Zarr data. It is defined as a Zarr store containing one or more Arrays and Groups.

A repository can be any valid Zarr hierarchy, ranging from a single Array to a deeply nested structure. A common use case is a single Zarr group containing multiple arrays that represent different physical variables sharing common spatiotemporal coordinates.

Key Design Principles:

• Object Storage Centric: Designed for cloud object storage without requiring an external database or catalog.
• Serializable Isolation: Reads are isolated from concurrent writes using committed snapshots. Writes are atomic and never partially visible.
• Time Travel: Previous snapshots remain accessible after new ones are written.
• Chunk Sharding: Decouples chunk storage from specific file names, allowing multiple chunks to be packed into a single object.
• Schema Evolution: Allows adding or removing Arrays and Groups with minimal overhead.

Icechunk uses Virtual Chunk Containers (VCCs) to allow for more efficient and flexible chunk referencing. Instead of storing full absolute URLs for every chunk in the manifest, you can use VCC names to create relative references.

VCC Names

• VCC names must be unique per repository (unless the name is not present).
• Authorization is still based on prefixes, not names, to maintain security.
• VCCs without a name can still exist, but they cannot be used for relative virtual chunk resolution.

Resolving Relative Chunk References

To use a relative reference, the location field in a ChunkRef must use the _vcc:// protocol. The hostname in this URL is interpreted as the VCC name, and the path is interpreted as being relative to the VCC's defined url_prefix.

Example Configuration:

s3:
  name: my-virtual-icechunk
  url_prefix: s3://testbucket/my-repo/chunks
  store: !s3_compatible
    region: us-east-1
    anonymous: false

Example Relative Reference: _vcc://my-virtual-icechunk/4K2JE645QXEXJ8BFDX70

This resolves to the absolute S3 path: s3://testbucket/my-repo/chunks/4K2JE645QXEXJ8BFDX70.

Absolute vs. Relative Locations

• Relative: Uses _vcc://<vcc_name>/<path>. The hostname must match a defined VCC name, otherwise dereferencing will error.
• Absolute: Uses standard protocols like s3://bucket/prefix/foo.nc. These must match one of the defined VCC prefixes to be valid.

The Icechunk Rust workspace is organized into several layered crates. Understanding this hierarchy helps in identifying which component to use for specific needs:

• icechunk-python: PyO3 bindings exposing the engine to Python (the main user-facing library).
• icechunk: The core storage engine handling transactions, version control, and repositories.
• icechunk-types: Shared foundational types like Path, ETag, Move, and error wrappers.
• icechunk-format: Handles binary format types and serialization (snapshots, manifests, transaction logs, repo info).
• icechunk-storage: Defines storage traits and common utilities.
• icechunk-arrow-object-store: A storage backend using Apache Arrow's object_store (supporting in-memory, local, GCS, Azure, etc.).
• icechunk-s3: An optional native AWS S3 storage backend.
• icechunk-macros: Procedural macro helpers for tests and internal use.

Virtual chunks allow an Icechunk repository to reference data stored in external locations (like S3 buckets).

Key Concepts:

• Multiple Locations: A single repository can contain virtual chunks hosted across different object stores, buckets, or cloud platforms.
• Credential Management: Accessing virtual chunks is a read-time concern. Because chunks may reside in different locations, users may need to provide multiple sets of credentials to fully access a repository. Icechunk is designed to provide ways to introspect a repository so users can identify which credentials are required to resolve all chunks.
• Data Integrity: If the underlying file hosting a virtual chunk changes after the reference is written, Icechunk aims to fail with a runtime error rather than serving potentially corrupted data, as offsets and lengths are typically not maintained during external file updates.

Icechunk must handle snapshot timestamps carefully during expiration to account for clock drift and ensure causality.

Challenges

• Clock Drift: Snapshots store creation timestamps from the user's clock, which may not be monotonic.
• Object Store Timestamps: Using Last Updated At from the object store is difficult because it changes during in-place updates.

Proposed Implementation Strategy

To maintain a reliable timeline for expiration, the following rules are used:

1. Enforce Monotonicity: A snapshot cannot be written unless its timestamp is strictly larger than its parent's timestamp.
2. Initial Write Validation: When a snapshot is written for the first time, its internal timestamp is compared against the object store's last-updated-at field. If the difference exceeds a few minutes, the flush process is interrupted, leaving the snapshot dangling (which is acceptable).
3. In-place Updates: When updating a snapshot in place, the original timestamp is maintained, and the value retrieved from the object store is ignored to prevent timestamp drift.