DOCSEARCHbyAlgolia
Submit Docs

IOMETE Documentation

website·Indexed 33 days ago

https://iomete.com/resources

Documentation for the IOMETE Kubernetes-native, self-hosted data lakehouse platform. Includes guides on deployment, DBT, Airflow, BI integrations, Git workflows, Iceberg Tables, Spark SQL, Pyspark, and SQL Quick Start.

Tokens
2.4K
Snippets
0
Records
38
Agent score
69%

What's inside IOMETE

  1. IOMETE Platform Overview

    IOMETE is a Kubernetes-native, self-hosted sovereign data lakehouse platform built on Apache Spark and Apache Iceberg. It is designed to run entirely within a customer's infrastructure (on-premises, private cloud, or public cloud VPC) to ensure data remains within the organization's trust perimeter. The platform includes components for compute, a SQL Editor, Spark jobs, and a Data Catalog.

  4. Deploy IOMETE on Kubernetes

    IOMETE is deployed within a Kubernetes environment. Key deployment requirements and steps include:

    • Infrastructure: Kubernetes cluster recommendations are provided in the setup guide.
    • Ingress: ISTIO is used as the ingress controller.
    • Storage: Minio is used for bucket preparation.
    • Backend Databases: Supports MySQL and PostgreSQL for managing internal and data lake metadata.
    • Networking: Custom DNS configuration is available for the Data Plane, and network policies can be configured for services.
    • Security: Custom Java Truststores (truststore.jks) can be used with custom CA certificates to prevent connection errors.

  5. Create and manage External Tables in IOMETE

    IOMETE allows the creation of external tables from various data sources. Supported formats and sources include:

    • File Formats: CSV, JSON, Parquet, and ORC.
    • Connectivity: JDBC for relational databases. Users can query these external sources and export data back to them.

  8. Integrate IOMETE with Apache Airflow

    IOMETE provides an Airflow Plugin that allows users to trigger Spark jobs. The IometeOperator is used to manage job parameters, configuration overrides, and track job run states. Run details can be passed through XCom.