dbt-adapters

This monorepo provides several packages categorized by their role in the dbt ecosystem:

Base and Testing

• dbt-adapters: The base adapter package.
• dbt-tests-adapter: The adapter integration test suite.

First-party Adapters

These are official adapters for specific data warehouses:

• dbt-athena
• dbt-bigquery
• dbt-postgres
• dbt-redshift
• dbt-snowflake
• dbt-spark

The dbt-athena adapter supports dbt version 1.7.* and includes the following capabilities:

Materializations

• Table: Supports both Hive tables (all engines) and Iceberg tables (Athena Engine v3 only). Iceberg tables require a unique table location.
• Incremental:
  • On Iceberg tables: Supports append strategy. The merge strategy is supported and is the only strategy that allows the use of unique_key.
  • On Hive tables: Supports insert_overwrite and append strategies. Note that unique_key is not supported for Hive tables.
• Seeds: Supported.
• Python Models: Supported (requires spark_work_group configuration).

Other Capabilities

• Correctly detects views and their columns.
• Supports snapshots.
• Supports threads for parallel execution.

The dbt-athena-community adapter provides the following capabilities:

• dbt Compatibility: Supports dbt version 1.7.*.
• Materializations:
  • Supports table materialization.
  • Iceberg tables: Supported only with Athena Engine v3 and require a unique table location.
  • Hive tables: Supported by both Athena engines.
  • Supports incremental models:
    • On Iceberg: Supports merge strategy (requires unique_key) and append strategy.
    • On Hive: Supports insert_overwrite and append strategies. Note: unique_key is not supported on Hive tables.
• Other Features:
  • Support for Python models (requires spark_work_group).
  • Support for seeds.
  • Support for snapshots.
  • Correct detection of views and their columns.

The RelationConfigChange class is used to model a change request, similar to a web request. It requires two main attributes:

• action: The type of operation being performed (e.g., 'create' is analogous to a GET, 'drop' is analogous to a DELETE).
• context: The information required to execute the change. In most scenarios, this is an instance of RelationConfigBase representing the desired new state of the relation.

The RelationConfig package provides an abstraction for inspecting existing relations and determining necessary changes. It is currently used to manage materialized views for Postgres and Redshift, as well as dynamic tables for Snowflake.

The package is built around three primary components:

1. RelationConfigBase: The base structure for representing a relation's state.
2. RelationConfigChange: Represents a specific action (like a change request) to be performed on a relation.
3. RelationConfigValidationMixin: A mixin used to add validation logic to the other classes.

The integration loading process follows these steps:

1. Startup: dbt-core loads the catalogs.yml file.
2. Config Creation: For every active write integration, dbt-core instantiates a CatalogIntegrationConfig object.
3. Adapter Callback: dbt-core calls the adapter's add_catalog_integration method, passing the config object.
4. Registration: The adapter uses the config to create and register a CatalogIntegration instance.

The BaseAdapter class is an abstract base class used as the foundation for all dbt adapter implementations. It provides a scaffold of methods designed to facilitate communication between dbt and a target database.

When developing or extending an adapter, you inherit from BaseAdapter. The specific methods you are required to implement will depend on the feature set and capabilities of your target database warehouse.

The S3 location for a table is determined by the following priority:

1. external_location (if defined).
2. s3_data_dir combined with s3_data_naming (if s3_data_dir is defined).
3. s3_staging_dir/tables/ (if no other directory is defined).

s3_data_naming options:

• unique: {s3_data_dir}/{uuid4()}/
• table: {s3_data_dir}/{table}/
• table_unique: {s3_data_dir}/{table}/{uuid4()}/
• schema_table: {s3_data_dir}/{schema}/{table}/
• schema_table_unique: {s3_data_dir}/{schema}/{table}/{uuid4()}/

Note: If your Athena workgroup has a default output location configured, these settings are ignored in favor of the workgroup's location.

When implementing Record/Replay, you do not need to record every interaction in full detail. To maintain security, you should exclude sensitive secrets from recordings.

Recommended exclusions:

• Parameters: Exclude passwords and authentication keys from the recorded parameters.
• Results: Exclude authentication tokens from the recorded results.

Since replay mode does not actually communicate with the warehouse, these secrets are not required for a successful replay.

The dbt-adapters library distinguishes between two primary adapter hierarchies:

1. BaseAdapter: Defines the fundamental functionality required for any adapter to interface with dbt-core. It contains a mix of methods with default implementations and abstract methods that concrete adapters must implement.
2. SQLAdapter: Inherits from BaseAdapter. It provides updated default implementations specifically tailored for SQL-based platforms and introduces additional required methods necessary for SQL support.

When building or extending an adapter, choose BaseAdapter for non-SQL platforms and SQLAdapter for SQL-based platforms.