DOCSEARCHbyAlgolia
Submit Docs

Delta Lake Documentation

website·Indexed 41 days ago

https://docs.delta.io/

Documentation for Delta Lake, an open source storage layer that brings ACID transactions to Apache Spark and big data workloads. Covers Delta Spark and Delta Kernel APIs, concurrency control, liquid clustering, change data feed, and connectors for Google BigQuery and AWS Athena. Includes guides on table batch reads and writes, schema validation, and version-specific features such as column mapping (1.2.0+), deletion vectors (2.3.0+), and default column values (3.1.0+).

Tokens
70K
Snippets
265
Records
431
Agent score
72%

What's inside Delta Lake

  1. Overview of Delta Lake type widening

    Type widening allows changing the type of columns in a Delta table to a wider type without rewriting the underlying Parquet files. This can be achieved through manual ALTER TABLE ALTER COLUMN commands or automatic type migration during schema evolution.

    Availability:

    • Preview: Delta Lake 3.2+
    • Fully Supported: Delta Lake 4.0+

  2. Overview of Delta Universal Format (UniForm)

    Delta Universal Format (UniForm) enables Delta tables to be read by Iceberg and Hudi clients. It works by automatically generating Iceberg or Hudi metadata asynchronously after a Delta commit, allowing a single set of Parquet data files to be accessed by clients of different formats with negligible write overhead.

  3. Overview of Delta Lake features

    Delta Lake is an open-source project used to build a Lakehouse architecture on top of existing data lakes (such as S3, ADLS, GCS, and HDFS). It provides the following core capabilities:

    • ACID Transactions: Ensures readers never see inconsistent data using serializable isolation levels on Spark.
    • Scalable Metadata Handling: Uses Spark's distributed processing to manage petabyte-scale tables with billions of files.
    • Unified Streaming and Batch: Tables function simultaneously as batch tables and streaming sources/sinks, supporting ingest, backfills, and interactive queries.
    • Schema Enforcement: Prevents the insertion of bad records by automatically handling schema variations during ingestion.
    • Time Travel: Provides data versioning for rollbacks, historical audit trails, and reproducible ML experiments.
    • Upserts and Deletes: Supports MERGE, UPDATE, and DELETE operations for change-data-capture (CDC) and slowly-changing-dimension (SCD) workflows.
    • Engine Compatibility: Supports various processing engines including Apache Spark, Apache Flink, Apache Hive, Apache Trino, and AWS Athena.

  4. Overview of Liquid Clustering in Delta Lake

    Liquid clustering is a data layout optimization feature available in Delta Lake 3.1.0 and above. It replaces traditional Hive-style partitioning and Z-Ordering by simplifying data layout decisions and allowing clustering columns to be redefined without rewriting the entire table. It is particularly beneficial for tables with high cardinality columns, significant data skew, rapidly growing datasets, or evolving access patterns.

  5. Overview of AWS Redshift Spectrum and Delta Lake integration

    AWS Redshift Spectrum can read Delta tables using a manifest file—a text file containing the list of data files (Parquet) to be read for a specific snapshot of the table. This integration is currently experimental and should be used with caution.

  6. Overview of Delta Change Data Feed (CDF)

    Change Data Feed (CDF) allows Delta tables to track row-level changes between versions. When enabled, the runtime records "change events" for all data written to the table, including the row data and metadata indicating if the row was inserted, deleted, or updated. This is useful for accelerating ETL/ELT operations in Silver and Gold tables, transmitting changes to downstream systems like Kafka or RDBMS, and maintaining audit trails.

  7. Overview of Delta Lake row tracking

    Row tracking allows Delta Lake to track row-level lineage by adding two hidden metadata fields to a table:

    • Row IDs: A unique identifier for each row that persists across MERGE or UPDATE operations.
    • Row commit versions: The table version in which the row was last modified. A new version is assigned whenever a row is modified via MERGE or UPDATE.

    Compatibility:

    • Available in Delta Lake 3.2.0+.
    • Enabling on existing non-empty tables is available in Delta Lake 3.3.0+.
    • Tables with row tracking use Delta Lake writer version 7. Protocol versions cannot be downgraded, and these tables are not writeable by clients that do not support the required writer protocol features.

  8. Overview of Delta Lake column mapping

    Column mapping allows Delta table columns and the underlying Parquet file columns to have different names. This enables schema evolution operations like renaming or dropping columns without rewriting the underlying Parquet files. It also allows the use of characters in column names that are not supported by Parquet, such as spaces, facilitating direct ingestion of CSV or JSON data. This feature is available in Delta Lake 1.2.0 and above.

  9. Overview of Delta Standalone

    Delta Standalone is a single-node Java library that allows JVM applications to read from and write to Delta tables without depending on Apache Spark. It implements the Delta Transaction Log Protocol to provide transactional guarantees.

    Important Note: Delta Standalone is deprecated. Users are encouraged to migrate to the Delta Kernel APIs.

    Key Constraints:

    • Memory: It runs in a single JVM and is limited by that JVM's memory. Users should configure the JVM to avoid Out-of-Memory (OOM) issues.
    • Parquet Writing: While it provides basic APIs for reading Parquet data, it does NOT include APIs for writing Parquet data. Developers must write Parquet files independently and then use Delta Standalone to commit those changes to the table.

  10. Overview of Delta Lake Catalog-Managed Tables

    Catalog-managed tables (introduced in Delta Lake 4.0.1) use a catalogManaged reader-writer feature that shifts the responsibility for commit atomicity and table discovery from the filesystem to the managing catalog. This enables advanced capabilities like multi-table transactions while maintaining ACID guarantees.

    Critical Constraint: Filesystem-based access is not supported for these tables. Delta clients must access them through a supporting catalog implementation rather than via direct path-based access to ensure consistency.

  11. Overview of Delta Connect (Spark Connect Support in Delta)

    Delta Connect adds Spark Connect support to Delta Lake for Apache Spark (available in Delta Lake 4.0.0 and above). It provides a decoupled client-server infrastructure using a gRPC layer, allowing Delta Lake operations to be executed from a remote client. This improves stability by running user code outside the Spark driver's JVM and simplifies integration with IDEs and non-JVM languages. Note: As of the current documentation, Delta Connect is in preview and not recommended for production workloads.

  12. Overview of Delta Lake table constraints

    Delta tables support standard SQL constraint management to ensure data quality and integrity. If a constraint is violated during a write operation, Delta Lake throws an InvariantViolationException and prevents the data from being added. Note that adding a constraint automatically upgrades the table writer protocol version.