DOCSEARCHbyAlgolia
Submit Docs

Delta Lake

repository·master·Indexed 11 days ago

https://github.com/delta-io/delta

An open-source storage framework that brings ACID transactions to data lakes to enable a Lakehouse architecture. It provides high-performance, reliable data storage and management for compute engines such as Spark, Flink, and Trino.

Tokens
173.6K
Snippets
370
Records
693
Agent score
94%

What's inside Delta Lake

  1. Overview of Delta Lake

    master
    Delta Lake is an open-source storage framework designed for Lakehouse architectures. It enables transactional guarantees (ACID) on top of data lakes and supports various compute engines including Apache Spark, PrestoDB, Flink, Trino, and Hive. It provides native APIs for Scala, Java, Rust, Ruby, and Python.

  2. Overview of Delta Lake capabilities

    master

    Delta Lake is an open-source storage layer that enables a Lakehouse architecture on top of existing data lakes (such as S3, ADLS, GCS, and HDFS). It provides the following core capabilities:

    • ACID Transactions: Ensures serializable isolation levels so readers never see inconsistent data.
    • Scalable Metadata: Uses Spark distributed processing to manage metadata for petabyte-scale tables.
    • Streaming and Batch Unification: A Delta table acts as both a batch table and a streaming source/sink, allowing for seamless streaming ingestion and batch backfills.
    • Schema Enforcement: Prevents the insertion of malformed records by automatically handling schema variations.
    • Time Travel: Enables data versioning for rollbacks, historical audits, and reproducible ML experiments.
    • DML Operations: Supports merge (upserts), update, and delete operations for complex use cases like Change Data Capture (CDC) and Slowly Changing Dimensions (SCD).
    • Connector Ecosystem: Supports reading and writing Delta tables via engines like Apache Spark, Apache Flink, Apache Hive, Apache Trino, and AWS Athena.

  3. Overview of the Delta Spark V2 Connector

    master

    The Delta Spark V2 Connector allows Apache Spark to read Delta tables by leveraging the Delta Kernel. It integrates Delta Kernel into Spark's query execution pipeline using Spark's DataSource V2 (DSV2) APIs.

    How it works

    The connector acts as a bridge between the Spark engine and Delta Kernel:

    1. Schema & Filtering: The Spark Driver requests the table schema and pushes down static and dynamic filters through the connector.
    2. Translation: The connector translates Spark requests into Delta Kernel APIs to fetch the table schema, push down filters, and retrieve the list of files to scan.
    3. State Resolution: Delta Kernel resolves the table state using the Catalog and Delta logs, applies file skipping, and returns the required files.
    4. Execution: The Spark Engine partitions the files (using a default of 128MB splits) and performs the actual Parquet scans via Spark's ParquetFileFormat.

  4. Overview of the Delta Benchmarking Framework

    master

    The Delta benchmarking framework is a tool designed to measure Delta Lake's performance. It is currently optimized to run benchmarks on Apache Spark clusters hosted on Amazon EMR or Google Cloud Dataproc, though it is designed to be extensible for other Spark-based environments.

    To use the framework, you must clone the repository locally, set up a Spark cluster (EMR or Dataproc), and execute the benchmark scripts provided in the benchmarks directory.

  6. What is Delta Kernel?

    master

    Delta Kernel is a set of libraries (available in Java and Rust) designed for building connectors that read from and write to Delta tables. It abstracts away the complexities of the Delta protocol, allowing developers to focus on data operations rather than protocol implementation.

    Key use cases include:

    • Reading data from small Delta tables in a single thread/process.
    • Reading data from large Delta tables using multiple threads in a single process.
    • Building connectors for distributed processing engines (e.g., Apache Spark, Apache Flink, Trino) to handle massive Delta tables.
    • Inserting data into Delta tables from single or distributed engines.

  7. Overview of the Delta Transaction Log Protocol

    master

    The Delta Transaction Protocol provides ACID properties to large collections of data stored in distributed file systems or object stores. It uses Multi-Version Concurrency Control (MVCC) to allow multiple writers to modify a table concurrently while providing readers with a consistent snapshot of the data.

    Core Design Goals

    • Serializable ACID Writes: Multiple writers can modify a table while maintaining ACID semantics.
    • Snapshot Isolation for Reads: Readers see a consistent snapshot even during concurrent writes.
    • Scalability: Supports billions of partitions or files with parallel query planning.
    • Self-describing: All metadata is stored alongside the data, eliminating the need for an external metastore and allowing tables to be moved using standard filesystem tools.
    • Incremental Processing: Readers can tail the Delta log to identify new data for efficient streaming.

    How it Works

    • Readers: Use the transaction log to selectively choose which data files to process, ensuring they see a single consistent snapshot.
    • Writers: Operate in two phases:
      1. Optimistic Write: Write new data files or updated copies of existing ones.
      2. Commit: Create the latest atomic version by adding a new entry to the log, recording which files to add/remove and any metadata changes.
    • Cleanup: Files no longer in the latest version are lazily deleted via the VACUUM command after a retention period (default 7 days).

  8. Overview of the Delta Flink Connector

    master

    The Delta Flink Connector allows Apache Flink streaming jobs to write data into Delta Lake tables.

    Key Characteristics:

    • Sink-only: Currently supports writing to Delta tables; no source support is available.
    • Exactly-once semantics: Achieved through integration with Flink checkpointing and a single global committer for atomic commits.
    • Memory Management: Includes features to limit concurrently open files to prevent OOM when writing to highly partitioned tables.
    • File Rolling: Provides controls for file size and count to mitigate the 'small-file problem'.
    • Architecture: Built on the Flink Connector V2 API and based on Delta Kernel.

  9. Use Delta Standalone for non-Spark Delta table interaction

    master

    Delta Standalone is a single-node Java library designed to read from and write to Delta tables without requiring Apache Spark. It implements the Delta Transaction Log Protocol to provide transactional guarantees. It is ideal for custom processing engines or applications that need low-level access to Delta Lake metadata.

    Important Note: Delta Standalone is deprecated. For new development, it is recommended to use the Delta Kernel APIs instead.

  10. Maintenance status of Connectors projects

    master

    Connectors projects are no longer being maintained in the master branch. Development and new releases have migrated to the Delta Kernel project.

    If you are using existing connectors, they will continue to receive support in maintenance mode via the spark-3.5-support branch.

  11. What is Delta Universal Format (UniForm)?

    master

    Delta Universal Format (UniForm) allows Delta Lake tables to be read by Iceberg and Hudi clients without duplicating data files.

    UniForm works by automatically generating Iceberg or Hudi metadata asynchronously after a Delta commit. This process happens on the same compute that completed the Delta transaction, ensuring negligible write overhead for Delta operations. A single copy of the Parquet data files provides access to all three formats (Delta, Iceberg, and Hudi).

  12. What is the vacuumProtocolCheck feature?

    master

    The vacuumProtocolCheck is a ReaderWriter feature designed to ensure that the VACUUM operation consistently performs both reader and writer protocol checks.

    Its primary purpose is to prevent data corruption that can occur when an older Delta Client executing a VACUUM command incorrectly deletes files that are still required by newer versions of Delta. By enforcing these checks, it ensures that protocol improvements or changes to the VACUUM process itself do not lead to accidental file deletion.