DOCSEARCHbyAlgolia
Submit Docs

Apache Hudi

repository·master·Indexed 11 days ago

https://github.com/apache/hudi

An open data lakehouse platform for ingesting, indexing, storing, and managing data across cloud environments using a high-performance open table format. Includes the hudi-agent-gateway, an AI gateway providing an agent loop, MCP server, and chat UI for lakehouse tools via Trino.

Tokens
151.1K
Snippets
384
Records
614
Agent score
91%

What's inside Apache Hudi

  2. Overview of the Hudi Integration Test Suite

    master

    The Hudi integration test suite is a long-running Spark job designed for end-to-end testing of Hudi datasets. It is used to improve release confidence and perform large-scale performance benchmarks.

    Key objectives include:

    • Testing compatibility with different versions of hdfs, parquet, spark, hive, and avro.
    • Generating diverse workloads by varying payload size, number of updates, number of inserts, and number of partitions.
    • Executing multiple operations such as insert, bulk_insert, upsert, compact, and query.
    • Supporting custom post-process actions and validations.

  3. Overview of Hudi Integration Bundles

    master

    The packaging directory contains modules designed to build fat/uber jars (bundles). These bundles enable Apache Hudi integration into various external systems by shading necessary dependencies into a single deployable unit.

    Key design principles for these bundles include:

    • System Alignment: Bundles aim to use the target system's existing jars and classes (e.g., using the target system's Parquet version instead of bundling a Hudi-specific one) to keep Hudi lightweight and flexible.
    • Dependency Management: Bundle POMs should only depend on required Hudi modules and modules declared as provided in parent POMs (like parquet-avro).
    • Shading and Inclusion: To include provided scoped dependencies in the bundle, they must be declared as compile dependencies in the bundle POM so the shade plugin can pull them in.
    • Explicit Whitelisting: All other runtime dependencies must be explicitly included using an <include> whitelist. New bundles should follow this pattern of explicit whitelisting and shading.

  4. Overview of hudi-agent-gateway

    master

    The hudi-agent-gateway is a deployable service designed to serve a Hudi lakehouse to AI agents. It provides three primary interfaces over a set of guarded lakehouse tools:

    1. Agent Chat API: A REST API (POST /v1/chat) that uses a LangGraph agent loop to provide grounded answers via multi-turn sessions. It supports optional SSE streaming.
    2. MCP Server: An HTTP-based Model Context Protocol server (/mcp) that allows external agents (like Claude) to call lakehouse tools directly.
    3. Chat UI: A first-party, dependency-free ChatGPT-style web interface accessible at /ui/.

    All queries are routed through Trino using guarded tools: query_lakehouse (read-only SQL), list_tables, and describe_table. The gateway enforces AST-level guardrails, ensuring only single SELECT statements are executed and injecting a LIMIT clause to prevent excessive data retrieval.

  5. Overview of hudi-lakehouse components and charts

    master

    The hudi-lakehouse repository provides Helm charts for decoupled lakehouse components. The architecture separates the deployable product from the local development scaffolding.

    Product Charts (charts/)

    • hudi-trino: Deploys Trino (server 472) with a built-in Hudi connector. It connects to your Hive Metastore/AWS Glue and your S3/GCS storage.
    • hudi-agent-gateway: Deploys the Hudi AI gateway (agent chat API, MCP server, and chat UI). It connects to hudi-trino and an LLM provider.
    • vllm: (Optional) Serves open-weight models via an OpenAI-compatible API. Requires a GPU.

    Local Development (local-dev/)

    A Minikube-based environment that stands up MinIO, a Derby-backed Hive Metastore, and the Spark operator to simulate a real lakehouse on a laptop.

  6. Overview of the Hudi Timeline UI

    master

    The Hudi Timeline UI is a lightweight, read-only web interface designed to visualize Hudi table metadata. It provides three primary views for a Hudi table:

    1. Timeline: A visual representation of the table's active timeline using vis-timeline. It shows completed actions as range bars (green), inflight actions as points (yellow), and requested actions as points (red). It maps pending actions like compaction, logcompaction, and clustering to their completed counterparts (commit, deltacommit, or replacecommit) to maintain consistency with Hudi's internal model.
    2. Table Config: Displays the table's hoodie.properties as a sorted JSON object.
    3. Schema History: Shows the current table schema and a history of schema changes over recent commits.

    The UI is served by the hudi-timeline-service module via a Javalin web application. The initial implementation runs in STANDALONE mode. A planned follow-up will support EMBEDDED mode within a Spark driver, including a dedicated tab in the Spark UI.

  7. Goals of the Expression Indexing Subsystem

    master

    The Expression Indexing implementation aims to achieve the following:

    • Modular & SQL-driven: A modular indexing subsystem that can be managed using full SQL support.
    • Unified Partitioning: Absorbing physical partitioning into the indexing layer and aggregating statistics at the storage partition level.
    • Engine Agnostic: Providing a mechanism that works across different query engines and languages.
    • Extensible: Supporting various index types including Bloom indexes, column stats, and future secondary indexes like vector indexes.

  8. Use the hudi-spark-datasource module for Spark integration

    master
    The hudi-spark-datasource module provides the Spark integration for Apache Hudi. It allows you to read and write Hudi tables using Spark SQL and DataFrames via a DataSource API. The module is architected in layers to support multiple Spark versions while providing version-specific optimizations.

  9. Hudi 1.0 Release Roadmap and Features

    master

    The Hudi 1.0 release focuses on evolving the storage format, introducing new APIs, and improving engine integrations. Key feature areas include:

    • Storage Format Evolution: Changes to the timeline, log files, file naming, metadata, and table properties.
    • New APIs & Abstractions: New interfaces for Table metadata, snapshots, index/metadata tables, key generation, and record merging.
    • Concurrency Control: A new non-blocking concurrency control protocol designed for Spark and Flink writers.
    • Indexing & Partitioning: Support for secondary indexes and logical partitioning via indexing (allowing different partitioning schemes on the same table via coarse indexes).
    • Cloud Optimization: Support for cloud-optimized storage layouts to mitigate cloud storage throttling issues.
    • Performance & Streaming: Improvements to writer performance (e.g., logging updates as deletes + inserts for overwrites) and streamlined Streaming CDC/Incremental reads.
    • Engine Integration: Enhanced support and performance for Spark, Flink, Presto, and Trino.

  10. Key features of Hudi Spark integration

    master

    The hudi-spark-datasource module provides several advanced capabilities for managing Hudi tables within Spark:

    • DataSource V1 Support: Full integration with Spark's DataSource API.
    • Spark SQL Integration: Native SQL support for Hudi tables via Spark Session extensions.
    • Stored Procedures: Built-in procedures for table management and operations.
    • Time Travel: Query historical versions of tables.
    • Incremental Queries: Efficient change data capture (CDC) reads.
    • Index Support: Includes Bloom filters, column statistics, record-level index, and partition stats.
    • Streaming Support: Structured Streaming source for continuous data ingestion.
    • CDC Support: Change Data Capture for tracking row-level changes.

  11. Overview of Partition Level Bucket Index (RFC-89)

    master

    The Partition Level Bucket Index is an evolution of the traditional Bucket Index. While the standard Bucket Index uses a fixed number of buckets for the entire table, the Partition Level Bucket Index allows users to set a specific number of buckets for different partitions using a rule engine (e.g., regular expression matching).

    Key Benefits

    • Flexibility: Avoids the limitation of a single fixed bucket count for all partitions.
    • Simplicity: Provides an alternative to complex automatic resizing (like consistent hashing) by allowing manual/offline reorganization via insert overwrite for specific partitions.
    • Seamless Upgrade: Existing Bucket Index tables can be upgraded to Partition-Level Bucket Index smoothly.

  12. What is the VARIANT type and when to use it

    master

    The VARIANT type is a data type designed for efficient storage and querying of semi-structured data (like JSON). It provides schema-on-read flexibility, allowing you to store evolving data structures without predefined schemas.

    Unlike storing JSON as a plain string, VARIANT uses an optimized binary encoding that enables fast navigation and element extraction without parsing the entire document. It offers the performance benefits of structured columns while maintaining the flexibility of a schema-less design.