dbt-codegen

When adding new functionality, you should follow a Test-Driven Development (TDD) workflow: create a failing test first, then implement the logic to make it pass.

Components of a new test

• A new seed file containing fixture data.
• A new model file to test against.
• A new test to assert the expected behavior.

Running specific tests

If you are working within a sub-project in the integration_tests folder, you can run only the relevant tests using dbt commands. Use the + operator in the --select flag to include all dependencies:

# Install dependencies for the sub-project
dbt deps --target {your_target}

# Run seeds, models, and tests for the selection
dbt build --target {your_target} --select +{your_selection_criteria}

Alternatively, you can use the project's make commands if you prefer running the full suite:

make dev target={your_target}
make test target={your_target}

To develop for dbt-codegen, you must set up a local environment including database credentials, a virtual environment, and the necessary dbt dependencies.

1. Configure Credentials

Set environment variables for your target database. For local Postgres development, generic values are provided in integration_tests/.env/. You can set these temporarily in your shell or permanently in your shell profile (e.g., ~/.bashrc or ~/.zshrc).

2. Set up a Database Target

It is recommended to use Postgres locally via Docker. You can spin up a Postgres container using make or docker-compose:

make setup-db
# or
docker-compose up --detach postgres

3. Set up a Virtual Environment

Create and activate a Python virtual environment in the root of the repository:

python3 -m venv .venv
source .venv/bin/activate

4. Install Dependencies

Upgrade pip and setuptools, then install dbt-core and the specific adapter for your target (e.g., postgres, redshift):

python3 -m pip install --upgrade pip setuptools

# Using make
make dev target=postgres

# Using pip
python3 -m pip install dbt-core dbt-postgres

# After installation, reload the environment
source .venv/bin/activate

# Summary of setup commands
python3 -m venv .venv
source .venv/bin/activate
make setup-db
make dev target=postgres
source .venv/bin/activate

The generate_unit_test_template macro generates the YAML configuration required for dbt unit tests for a specific model. The generated YAML includes all upstream references as given inputs (including their columns) and placeholders for the expect output columns.

To use this macro:

1. Create your model with its original SQL.
2. Call the macro as a dbt operation.
3. Copy the generated YAML from the command line logs.
4. Create a new YAML file (or update an existing one) with the logged code.
5. Populate the rows sections with your test data for both given inputs and expect outputs.

Arguments:

• model_name (required): The name of the model for which to generate the template.
• inline_columns (optional, default=False): If set to True, attempts to put all columns on the same line in the YAML output.

$ dbt run-operation generate_unit_test_template --args '{"model_name": "order_items", "inline_columns": true}'

The generate_source macro creates lightweight YAML for dbt Sources, which can be pasted into a schema file.

Arguments

• schema_name (required): The schema name containing source data.
• database_name (optional, default=target.database): The database for the source data.
• table_names (optional, default=none): A list of specific tables to generate definitions for.
• generate_columns (optional, default=False): Whether to include column names.
• include_descriptions (optional, default=False): Whether to include description placeholders.
• include_data_types (optional, default=True): Whether to include data types for columns.
• table_pattern (optional, default=%): A prefix/postfix to subselect tables within a schema.
• exclude (optional, default=''): A string to exclude from selection.
• name (optional, default=schema_name): The name of the source.
• include_database (optional, default=False): Whether to add the database to the definition.
• include_schema (optional, default=False): Whether to add the schema to the definition.
• case_sensitive_databases (optional, default=False): Match database case (not compatible with Redshift).
• case_sensitive_schemas (optional, default=False): Match schema case (not compatible with Redshift).
• case_sensitive_tables (optional, default=False): Match table case (not compatible with Redshift).
• case_sensitive_cols (optional, default=False): Match column case.

Outputting to a file

You can pipe the output directly to a file using the --quiet flag:

dbt --quiet run-operation generate_source --args '{"table_names": ["orders"]}' > models/staging/jaffle_shop/_sources.yml

{{ codegen.generate_source('raw_jaffle_shop') }}