delta-rs

Delta Lake provides performance optimizations over standard data lakes (like Parquet or CSV) through two primary mechanisms:

1. File Skipping: Delta Lake stores min/max values for each column of every file in the transaction log. This allows queries to skip entire files based on metadata without opening the files themselves.
2. Efficient File Discovery: Instead of performing slow file-listing operations on cloud object stores (which can be particularly slow with Hive-style partitions), Delta Lake retrieves all necessary file paths directly from the transaction log.

To further optimize performance, you can use techniques like partitioning or Z-Ordering to colocate similar data in the same files, maximizing the effectiveness of file skipping.

A Delta table is composed of two main parts:

1. Parquet files: These contain the actual data.
2. _delta_log directory: This contains a transaction log that stores metadata about all transactions, including:
   • Files added to the table.
   • The schema of the files.
   • Column-level metadata (e.g., min/max values for each file).

This architecture allows for ACID transactions by using a transaction log to manage state rather than relying on the physical presence of files in storage.

Delta Lake is highly portable and supports a wide range of environments and languages:

• Languages: This project provides native APIs for Rust and Python without requiring a Java or Scala dependency. It serves as an alternative to using pandas, Polars, DuckDB, or DataFusion directly on raw files.
• Cloud Providers: Full support for AWS, GCP, and Azure.
• Local/On-Prem: Can be run on local machines or in on-premises environments.

Delta Lake implements file skipping by storing min/max metadata for each column in the transaction log. Query engines use this metadata to skip entire files that do not contain relevant data for a specific predicate (e.g., WHERE age < 20), significantly reducing I/O.

Key requirements for file skipping:

• File Format: Data must be in a format that supports file-level metadata, such as Parquet. Formats like CSV do not support this optimization.
• Query Type: Queries must include predicates (filters) that can be evaluated against the min/max values. Queries like GROUP BY without filters typically cannot benefit from file skipping.
• Data Layout: The effectiveness of skipping depends on how data is distributed across files.

You can access any storage service supported by Apache OpenDAL using a generic backend in delta-rs. This allows you to use services that do not have a dedicated native integration in delta-rs by using an OpenDAL scheme.

In the Python wheel, the following services are enabled via OpenDAL:

• fs (filesystem)
• memory
• s3
• gcs (Google Cloud Storage)
• azblob (Azure Blob Storage)
• azdls (Azure Data Lake Storage)
• oss (Alibaba Cloud OSS)
• obs (Huawei Cloud OBS)
• cos (Tencent Cloud COS)
• tos (ByteDance TOS)
• b2 (Backblaze B2)
• swift (OpenStack Swift)
• webhdfs
• webdav
• ftp
• sftp
• hf (HuggingFace Hub)

Unlike traditional data lakes where only appending data is straightforward, Delta Lake supports a full range of data manipulation operations (DML) performed efficiently under the hood:

• Appends: Adding new data to the table.
• Upserts: Updating existing records or inserting new ones.
• Deletes: Removing specific rows of data.
• Replace where: Replacing a subset of data based on a predicate.

Delta Lake uses transactions to ensure that write operations are safe and do not cause downtime or data corruption. Transactions in Delta Lake guarantee that operations:

• Are Atomic: They either finish completely or do not run at all.
• Are Isolated: They are executed in a serial manner and do not conflict with other transactions.
• Maintain Integrity: They do not corrupt the table or violate table constraints.

This prevents common data lake issues such as schema mismatches during appends, reading incorrect/partial data during active writes, and data loss from conflicting concurrent transactions.