DOCSEARCHbyAlgolia
Submit Docs

DuckLake Documentation

repository·main·Indexed 25 days ago

https://github.com/duckdb/ducklake

DuckLake is an open Lakehouse format built on SQL and Parquet that uses a catalog database for metadata and Parquet files for data storage. It provides a DuckDB extension for direct read/write access, supporting features such as time travel via the AT (VERSION => <version_number>) clause and change data feeds using the table_changes function. The system supports S3-compatible storage (e.g., MinIO) and provides a DuckLakeMetadataManager for managing snapshots, tables, and data files.

Tokens
3.6K
Snippets
11
Records
22
Agent score
83%

What's inside DuckLake

  2. Configure MinIO alias and create a bucket

    main

    Use the MinIO client (mc) to set an alias for your local server and create a bucket for DuckLake storage.

    1. Set the alias: mc alias set '<alias_name>' '<endpoint>' '<access_key>' '<secret_key>'.
    2. Create the bucket: mc mb <alias_name>/<bucket_name>.

    Example using default credentials and endpoint 10.1.0.20:9000:

    mc alias set 'myminio' 'http://10.1.0.202:9000' 'minioadmin' 'minioadmin'
mc mb myminio/demo-ducklake-minio-bucket

  3. Install DuckDB and MinIO dependencies

    main

    To run the MinIO demo, install the DuckDB CLI and the MinIO server/client tools using Homebrew. These instructions were tested with DuckDB v1.3.0.

    # Install DuckDB CLI
brew install duckdb

# Install MinIO server and client (mc)
brew install minio/stable/minio
brew install minio/stable/mc
    brew install duckdb
brew install minio/stable/minio
brew install minio/stable/mc

  4. Use DuckLake in DuckDB

    main

    DuckLake databases are accessed using the ATTACH syntax. You must specify a metadata file (the catalog database) and a DATA_PATH where the Parquet files will be stored. Once attached, you can interact with DuckLake tables using standard SQL.

    ATTACH 'ducklake:metadata.ducklake' AS my_ducklake (DATA_PATH 'file_path/');
USE my_ducklake;

-- Create and populate a table
CREATE TABLE my_ducklake.my_table(id INTEGER, val VARCHAR);
INSERT INTO my_ducklake.my_table VALUES (1, 'Hello'), (2, 'World');

-- Query the table
FROM my_ducklake.my_table;

  5. Set up a local MinIO S3 server

    main

    Start a local MinIO server to mock S3 services. MinIO will host data and metadata in the specified folder.

    1. Create a directory for the data: mkdir -p path/to/some_folder.
    2. Start the server: minio server path/to/some_folder.

    Note: If the default port is in use, customize it using the --address ':NUMBER' flag.

    Important: Record the HTTP endpoint, port, and credentials (default is often minioadmin/minioadmin) provided in the server output.

    mkdir -p path/to/some_folder
minio server path/to/some_folder

  6. Install the DuckLake extension

    main

    You can install the DuckLake extension in DuckDB using the INSTALL command. To install the latest development version, use FORCE INSTALL from the core_nightly repository.

    INSTALL ducklake;

-- To install the latest development version
FORCE INSTALL ducklake FROM core_nightly;

  7. Build the DuckLake extension

    main

    To build the extension from source, initialize and update submodules, then use make. For faster builds using multiple cores, use make GEN=ninja release.

    git submodule init
git submodule update
# For multi-core builds:
# make GEN=ninja release
make pull
make

  8. Run MinIO and MinIO Client via Docker Compose

    main

    You can use the provided docker-compose.yml to set up a local MinIO environment for testing DuckLake. This configuration spins up a MinIO server and a MinIO Client (mc) container that automatically initializes a bucket named mybucket.

    Service Details:

    • MinIO Server (minio):
      • S3 API Port: 9000
      • Web Console Port: 9001
      • Root User: admin (via MINIO_ROOT_USER)
      • Root Password: password (via MINIO_ROOT_PASSWORD)
      • Timezone: UTC
    • MinIO Client (mc):
      • Automatically waits for the MinIO server to be ready.
      • Configures an alias local pointing to http://minio:9000.
      • Ensures the bucket mybucket exists by running mc mb local/mybucket if it is missing.
    version: "3.8"

services:
  minio:
    image: minio/minio
    container_name: minio
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=password
      - TZ=UTC
    networks:
      - s3_net
    volumes:
      - ./data:/data
    ports:
      - "9000:9000"
      - "9001:9001"
    command: ["server", "/data", "--console-address", ":9001"]

  mc:
    depends_on:
      - minio
    image: minio/mc
    container_name: mc
    networks:
      - s3_net
    entrypoint: >
      /bin/sh -c "
      until /usr/bin/mc alias set local http://minio:9000 admin password; do
        echo 'Waiting for MinIO...';
        sleep 2;
      done;

      if ! /usr/bin/mc ls local/mybucket > /dev/null 2>&1; then
        /usr/bin/mc mb local/mybucket;
      fi

      echo 'MinIO bucket initialized.';
      tail -f /dev/null
      "

networks:
  s3_net:

  9. Use DuckLake with MinIO S3 storage in DuckDB

    main

    Follow these steps in the DuckDB CLI to configure S3 access and attach a DuckLake database backed by an S3 bucket.

    1. Configure S3 Secret: Create a secret to allow DuckDB to communicate with MinIO.
    2. Attach DuckLake: Use the ducklake: prefix to attach a local file that uses the S3 bucket as its DATA_PATH.
    3. Operate on Data: Once attached, you can create tables, perform deletions, and inspect files in the bucket using glob functions.
    --- Setup relevant DuckDB temporary secret to gain access to the local MinIO S3 bucket
create secret (type s3, key_id 'minioadmin', secret 'minioadmin', endpoint '10.1.0.202:9000', use_ssl false, url_style 'path');

--- Attach a local DuckDB file (minio-ducklake-demo.ducklake) and a local (but using S3 protocol) bucket
ATTACH 'ducklake:minio-ducklake-demo.ducklake' as db (DATA_PATH 's3://demo-ducklake-minio-bucket');

--- Use the just attached DuckLake as default Database
USE db;

--- Create a table with some data (in the ducklake)
CREATE TABLE numbers AS (SELECT random() FROM range(100000));

--- Check which files are in the bucket
FROM glob('s3://demo-ducklake-minio-bucket/**');

--- Remove some data
DELETE FROM numbers WHERE #1 < 0.1;

--- Check which files are in the bucket, there should now also be a delete file
FROM glob('s3://demo-ducklake-minio-bucket/**');

  11. Use Change Data Feed in DuckLake

    main

    You can retrieve changes to a table using the table_changes function. This returns a result set containing the snapshot_id, rowid, change_type (e.g., 'insert'), and the row data.

    FROM my_ducklake.table_changes('my_table', 2, 2);

  12. Configure DuckLake catalog options

    main

    The DuckLakeCatalog allows configuring various parameters via DuckLakeOptions. Key configuration properties include:

    • metadata_database: The name of the metadata database.
    • metadata_schema: The schema used for metadata.
    • metadata_path: The filesystem path to the metadata.
    • data_path: The filesystem path to the data files.
    • metadata_type: The type of metadata server being used.
    • encryption: The encryption setting (e.g., DuckLakeEncryption::ENCRYPTED).

    You can also set a custom path separator (defaults to /) using Separator().