DOCSEARCHbyAlgolia
Submit Docs

Apache Hudi Documentation

website·Indexed 20 days ago

https://hudi.apache.org/

An open source data lakehouse platform providing database-like capabilities, including ACID transactions and record-level upserts.

Tokens
624
Snippets
0
Records
7
Agent score
100%

What's inside Apache Hudi

  1. Overview of Apache Hudi capabilities

    Apache Hudi is an open-source data lakehouse platform that provides database-like capabilities on top of cloud object storage or HDFS. Key features include:

    • ACID Transactions: Ensures data integrity for concurrent operations.
    • Record-level Upserts and Deletes: Allows updating or deleting individual records within large datasets.
    • Indexing: A multi-modal subsystem including bloom filters, record-level indexes, secondary indexes, expression indexes, and vector search.
    • Change Data Capture (CDC) & Incremental Processing: Enables processing only changed data rather than full table re-computations.
    • Storage Types: Supports Copy-on-Write (CoW) and Merge-on-Read (MoR) storage.
    • File Format Support: Manages tables using Apache Parquet, Apache ORC, Lance, and Apache Avro.
    • Table Services: Built-in automation for compaction, clustering, and cleaning.
    • Interoperability: Can interoperate with Apache Iceberg and Delta Lake metadata via Apache XTable.
    • Query Engine Support: Compatible with Apache Spark, Apache Flink, Presto, Trino, and Hive.

  3. Implement Change Data Capture (CDC) patterns on the lakehouse

    Apache Hudi supports Change Data Capture (CDC) patterns, allowing users to stream changes from operational databases into the data lakehouse while maintaining consistency and record-level updates.

  4. Choose between Copy-on-Write (CoW) and Merge-on-Read (MoR) table types

    Apache Hudi provides two primary table types to handle data updates and deletes: Copy-on-Write (CoW) and Merge-on-Read (MoR). CoW creates new versions of files on every update, while MoR logs updates to delta files and merges them during read or compaction, making it more suitable for high-frequency write workloads.

  5. Migrate existing Parquet or Hive tables to Hudi

    Apache Hudi supports migrating existing Parquet or Hive tables into Hudi tables. This can be achieved through 'bootstrapping', which allows the migration to occur without needing to rewrite all existing data.

  7. Accelerate upserts and point lookups using the Record Level Index

    The Record Level Index in Apache Hudi is designed for large-scale datasets to provide blazing fast indexing, significantly accelerating the performance of upserts and point lookup operations by reducing the amount of data scanned to locate specific records.