DOCSEARCHbyAlgolia
Submit Docs

OLake

website·Indexed 33 days ago

https://olake.io

An open-source data replication and Apache Iceberg table maintenance platform for moving data from databases and Kafka to lakehouses. Documentation covers Go EL engines, installation, configuration, sources, destinations, and architecture.

Tokens
944
Snippets
1
Records
20
Agent score
69%

What's inside OLake

  1. Overview of OLake Fusion Table Maintenance

    OLake Fusion is an open-source tool used to automate Apache Iceberg table maintenance. Its primary function is the compaction of small files to optimize table performance. According to benchmarks, OLake Fusion is approximately 2x faster and 50% cheaper than Spark for Iceberg compaction tasks.

  2. Overview of OLake Go EL Engine

    OLake Go is an open-source Extract and Load (EL) engine designed to replicate data from various databases and streaming sources into Apache Iceberg tables or Parquet files on object storage. It supports three primary synchronization modes: full refresh, incremental sync, and Change Data Capture (CDC) with stateful, resumable syncs. It is compatible with major Iceberg catalogs (AWS Glue, Hive Metastore, REST catalogs like Nessie, Polaris, and Unity Catalog) and can be queried by engines such as Trino, Spark, Snowflake, Databricks, and ClickHouse.

  3. Understand OLake high-throughput replication optimizations

    OLake implements specific core optimizations to achieve high-throughput data replication, ensuring efficient movement of data from sources like databases and Kafka to lakehouses.

  5. Replicate Kafka topics to Apache Iceberg

    OLake supports Kafka as a source for Iceberg ingestion, providing capabilities for synchronization, concurrency management, and partition handling.

  6. Handle schema evolution in Iceberg pipelines

    OLake leverages Apache Iceberg's field ID-based column tracking to manage schema evolution. This allows the pipeline to handle added, dropped, renamed, and widened columns without causing pipeline failures or data loss.

  8. Synchronize Microsoft SQL Server data to a lakehouse using OLake

    OLake provides a guided process for synchronizing data from Microsoft SQL Server (MSSQL) to a lakehouse destination, enabling data replication from the database to the lakehouse environment.

  10. Contribute to the OLake open-source project

    OLake is an open-source project licensed under Apache 2.0. Developers can contribute to OLake Go by following the contribution guide, which includes instructions for setting up the development environment. The project is hosted on GitHub.
    # OLake GitHub Repository
https://github.com/datazip-inc/olake