DOCSEARCHbyAlgolia
Submit Docs

Apache Gravitino Documentation

repository·main·Indexed 25 days ago

https://github.com/apache/gravitino

Documentation for Apache Gravitino, including the Gravitino CLI, Gvfs-fuse (a Fuse client for mounting filesets), and the Hologres JDBC Catalog. It provides guides on building the CLI, configuring Gvfs-fuse via TOML, running JCStress-based concurrency tests, and using the gravitino-release skill for release pipelines.

Tokens
334.8K
Snippets
627
Records
1.4K
Agent score
84%

What's inside Apache Gravitino

  1. Overview of Iceberg REST Catalog Service

    main

    The Apache Gravitino Iceberg REST server implements the Apache Iceberg REST API specification. It is specifically designed to manage Iceberg tables via the Iceberg REST API, whereas the main Gravitino server manages multiple table types (JDBC, Hive, Iceberg, Hudi, Paimon, etc.) via Gravitino unified interfaces.

    Access the Iceberg REST endpoint at: http://$ip:$port/iceberg/.

  2. Overview of the Gravitino Job System

    main

    Introduced in version 1.0.0, the Gravitino job system allows users to register, run, and manage jobs through a unified interface. It enables 'metadata-driven actions,' such as compacting Iceberg tables or cleaning data based on TTL properties, by integrating with existing metadata management.

    Key Characteristics:

    • Unified Job Submitter: Gravitino acts as a submitter rather than an execution engine. It does not provide actual execution capabilities.
    • Extensible Executors: It relies on external job executors (schedulers) like Apache Airflow or Apache Livy to perform the actual work.
    • Current Limitations: As of the current development phase, the system supports running a single job at a time and does not yet support job scheduling.

  3. Overview of Fileset Catalog

    main

    The Fileset Catalog uses a Hadoop Compatible File System (HCFS) to manage the storage locations of filesets. It is designed to provide a unified way to manage file-based storage through Gravitino.

    Supported storage backends include:

    • Local filesystem
    • HDFS
    • S3
    • GCS
    • OSS
    • Azure Blob Storage (ADLS)
    • Tencent Cloud COS (supported since version 2.0.0)

  4. Overview of the Gravitino Trino Connector

    main

    The Gravitino Trino connector allows Trino to interact with catalogs managed by Apache Gravitino. This enables Trino to access diverse data sources (such as Hive, Iceberg, MySQL, PostgreSQL, and AWS Glue) through a unified interface provided by Gravitino.

    Key areas for users include:

    • Setup: Requirements, Installation, and Configuration.
    • Security: Authentication methods.
    • Data Access: Supported catalogs and SQL capabilities.
    • Advanced Features: Cascading queries and User Defined Function (UDF) support.

  6. Overview of Spark Connector for AWS Glue Catalog

    main

    The Apache Gravitino Spark connector allows for seamless federated queries by enabling access to and management of metadata within the AWS Glue Data Catalog. It supports most DDL and DML operations in SparkSQL, with the following exceptions:

    • Function operations (though Gravitino UDFs are supported)
    • Partition operations
    • View operations
    • LOAD clause
    • CREATE TABLE LIKE clause
    • TRUNCATE TABLE clause

  7. Overview of Table Maintenance Service (Optimizer)

    main

    The Table Maintenance Service (Optimizer) automates table maintenance by integrating statistics/metrics collection, rule evaluation/strategy recommendation, and job template-based execution.

    Current Alpha Limitations:

    • Operated via the optimizer CLI workflow.
    • Built-in maintenance is focused on Iceberg table compaction.
    • Compaction is currently limited to Iceberg tables using identity partition transforms.

  8. Overview of Gvfs-fuse

    main
    Gvfs-fuse is a Fuse client for Gravitino filesets. It enables users to mount Gravitino filesets (including S3 filesets) to a local file system via Fuse. Once mounted, fileset files can be accessed, read, written, and manipulated (including directory operations and file attribute changes) as if they were part of the local file system.

  9. Overview of Apache Gravitino Release Manager

    main
    The Apache Gravitino Release Manager is a staged pipeline designed to guide users through the full release process. It uses specialized scripts located in the dev/release/ directory of the repository to manage version tagging, artifact building, documentation, publishing, and finalization. The process is stateful, tracking progress via .done files to prevent accidental re-runs of completed stages.

  10. Overview of Lance REST Service

    main

    The Lance REST service provides a RESTful interface for managing Lance datasets via HTTP endpoints. Introduced in Gravitino version 1.1.0, it acts as a bridge between applications (Python/Java) and Lance datasets (stored on S3, GCS, or local file systems). It uses the Gravitino Client API to interact with the Gravitino Server for metadata management and unified governance.

    Key features include:

    • Full compliance with the Lance REST API specification.
    • Support for namespace and table management.
    • Ability to run standalone or integrated with a Gravitino server.
    • Metadata storage in Gravitino for unified governance.

  11. Use the Daft Connector for Gravitino

    main

    The Daft connector integrates the Daft distributed dataframe library with Apache Gravitino's unified metadata management system. It allows Daft users to access tables and filesets managed by Gravitino, leverage Gravitino's unified metadata layer across various data sources, and utilize automatic credential management for cloud storage systems.

    For detailed installation, configuration, and API references, refer to the official Daft documentation.