dbt-databricks Adapter

The dbt seed command follows a specific execution lifecycle to load CSV data into Databricks. The project has transitioned from V1 to V2. The primary difference in V2 is the removal of unsupported operations, specifically the creation of indexes and the explicit commit step, resulting in a cleaner execution path.

V1 Seed Flow Lifecycle

1. In-memory Preparation: Creates an in-memory table from the CSV and stores the result.
2. Prehooks: Executes prehooks (first with inside_transaction=False, then with inside_transaction=True).
3. Table Existence Check:
   • If the table does not exist: Creates the table and performs chunked inserts.
   • If the table exists:
     • If it is not a table: Raises a compiler error.
     • If it is a table and is a Delta table: Performs create or replace table... followed by chunked inserts.
     • If it is a table but not Delta: Drops the existing table, creates a new one, and performs chunked inserts.
4. Post-processing: Applies grants, creates indexes (noted as a legacy/questionable step in V1), runs posthooks (inside_transaction=True), commits the transaction, and finally runs posthooks (inside_transaction=False).

V2 Seed Flow Lifecycle (Current)

V2 simplifies the process by removing unsupported calls:

1. In-memory Preparation: Creates an in-memory table from the CSV and stores the result.
2. Prehooks: Executes prehooks (first with inside_transaction=False, then with inside_transaction=True).
3. Table Existence Check:
   • If the table does not exist: Creates the table and performs chunked inserts.
   • If the table exists:
     • If it is not a table: Raises a compiler error.
     • If it is a table and is a Delta table: Performs create or replace table... followed by chunked inserts.
     • If it is a table but not Delta: Drops the existing table, creates a new one, and performs chunked inserts.
4. Post-processing: Applies grants and runs posthooks (inside_transaction=True then inside_transaction=False).

The incremental materialization in dbt-databricks follows two distinct execution paths depending on whether a relation already exists in the target schema.

Existing Incremental Flow

When a relation already exists and is not being replaced (i.e., not a full refresh or a type change), dbt-databricks performs the following:

1. Pre-hooks: Runs any configured pre-hook commands.
2. Intermediate Materialization: Creates an intermediate version of the model using either SQL or Python.
3. Schema Processing: Detects and processes schema changes.
4. Merge Logic: Applies the merge logic to integrate the intermediate data into the target table.
5. Post-processing: Applies configuration changes (like Liquid clustering, tags, or table properties), persists documentation, applies grants, runs optimize, and finally runs post-hook commands.

New Incremental Flow

When a relation does not exist, or when a full refresh/type change triggers a replacement, the flow involves creating a staging environment to ensure data integrity:

1. Staging: Creates a staging table based on the model schema.
2. Constraints & Metadata: Applies check constraints, tags, and table properties to the staging table.
3. Data Loading: Inserts the intermediate materialization into the staging table.
4. Atomic Swap: Uses a rename pattern (renaming the existing table to a backup and the staging table to the target) to swap the new data into place.
5. Cleanup: Drops the backup and applies final configurations (Liquid clustering, grants, etc.) to the new target table.

To test macros that depend on dbt configuration, variables, or adapter state, you must manipulate the test context.

Config Manipulation

Use this to test macros that read model configuration (e.g., file_format, tblproperties, or liquid_clustering).

Variable Manipulation

Use this to test macros that use var() to read dbt project variables, such as feature flags (e.g., DATABRICKS_SKIP_OPTIMIZE).

Context Mocking

Use this to isolate the macro under test by mocking dbt built-in functions or adapter methods (e.g., adapter.get_relation or adapter.run_query).

def test_macro_with_config(self, config, template_bundle):
    """Test how macros respond to different model configurations"""
    config["tblproperties"] = {"key": "value"}
    config["file_format"] = "delta"
    config["liquid_clustering"] = ["col1", "col2"]

    result = self.run_macro(template_bundle.template, "create_table_as",
                           template_bundle.relation, "select 1")
    assert "tblproperties" in result
    assert "delta" in result.lower()

def test_macro_with_variables(self, var, template_bundle):
    """Test macros that use dbt variables for feature flags"""
    var["DATABRICKS_SKIP_OPTIMIZE"] = True
    result = self.run_macro(template_bundle.template, "optimize", template_bundle.relation)
    assert result == ""

def test_macro_with_context_mocks(self, template_bundle):
    """Mock external dependencies to isolate macro behavior"""
    template_bundle.context["get_columns_in_query"] = Mock(return_value=[])
    template_bundle.context["adapter"].get_relation = Mock(return_value=None)
    
    result = self.run_macro(template_bundle.template, "my_macro", template_bundle.relation)

The V2 execution model introduces more sophisticated logic for updating existing views, specifically utilizing an update_via_alter mechanism to minimize disruption. The flow depends on whether the relation exists, whether it is a view, and whether the update_via_alter configuration is active.

V2 Execution Sequence:

1. Run pre-hooks.
2. Check if an existing relation exists:
   • If no: Create the view, apply tags, apply grants, and run post-hooks.
   • If yes: Check if the existing relation is a view and update_via_alter is enabled:
     • If no: Use the Replace Flow (see replace_flow.md), then apply tags, grants, and post-hooks.
     • If yes: Check if the view matches the project definition:
       • If matches: Apply grants, then post-hooks.
       • If does not match: Use ALTER to update the view, then apply grants and post-hooks.
3. If using ALTER, apply grants and post-hooks.

The V1 Table Flow describes the sequential execution steps dbt-databricks takes when materializing a model. The lifecycle follows this general order:

1. Pre-execution: Runs pre-hooks.
2. Relation Management: Checks if an existing relation exists and if it is replaceable. If not replaceable, it drops the existing relation.
3. Model Creation:
   • If the language is Python, it creates the table using Python.
   • If the language is SQL, it checks if the table is a Delta table. If yes, it uses create or replace table...; otherwise, it uses create table....
4. Post-creation Metadata:
   • Applies grants.
   • If the language is Python, it applies tblproperties via ALTER.
   • If the language is SQL, it applies tags via ALTER.
   • Applies docs (persisting documentation) via ALTER.
   • Applies constraints via ALTER.
5. Optimization & Post-execution: Runs OPTIMIZE and then executes post-hooks.