DOCSEARCHbyAlgolia
Submit Docs

dbt-external-tables

repository·main·Indexed 18 days ago

https://github.com/dbt-labs/dbt-external-tables

A dbt package that allows the definition of external tables in S3 or GCS using dbt YAML configurations, enabling queries of files directly from cloud storage as native database tables.

Tokens
536
Snippets
2
Records
2
Agent score
14%

What's inside dbt-external-tables

  1. Understand the integration test data sources

    main

    The integration tests for dbt-external-tables use public datasets stored in S3 and Google Cloud Storage to verify that external tables are correctly staged across various databases, file formats, and partitioning schemes. The tests validate that the final combined output matches the reference file seeds/people.csv.

    Public storage buckets:
- s3://dbt-external-tables-testing
- gs://dbt-external-tables-testing/

  2. Transform and rewrite Parquet files using Polars

    main

    This guide demonstrates how to read existing Parquet files, rename columns to a capitalized format using the polars library, and write the transformed data back to a new file path. This pattern is useful for preparing data structures before defining external tables in dbt.

    To perform this transformation:

    1. Read the source Parquet file using pl.read_parquet().
    2. Use .rename() with a dictionary mapping old column names to new column names.
    3. Ensure the destination directory exists using os.makedirs().
    4. Write the resulting DataFrame to the new path using .write_parquet().
    import polars as pl
import os

# Define column mapping
new__col_dict = {'id':'Id', 'first_name':'First_Name', 'last_name':'Last_Name', 'email':'Email'}
fpath = 'integration_tests/public_data/parquet{}/section={}/people_{}.parquet'

# Prepare destination
dest_path = fpath.format('_capitalized', 'a', 'a')
os.makedirs(os.path.dirname(dest_path), exist_ok=True)

# Read, transform, and write
df_a = pl.read_parquet(fpath.format('', 'a', 'a')).rename(new__col_dict)
df_a.write_parquet(dest_path)