DOCSEARCHbyAlgolia
Submit Docs

dbterd

repository·main·Indexed 43 days ago

https://github.com/datnguye/dbterd

A tool that generates Entity Relationship Diagrams (ERDs) as code from dbt artifacts. It transforms dbt metadata into formats including DBML, Mermaid, PlantUML, GraphViz, D2, DrawDB, and JSON. It supports relationship detection via dbt tests, Semantic Layer entity definitions, and model contract foreign_key constraints (dbt 1.9+). Features include a Python API via the DbtErd class, CLI configuration via YAML or TOML, and integration with both dbt Core and dbt Cloud.

Tokens
29.7K
Snippets
87
Records
118
Agent score
55%

What's inside dbterd

  1. What is a Target Adapter?

    main

    A Target Adapter is a translator component in dbterd that transforms parsed dbt artifacts (tables and relationships) into a specific ERD output format. It allows you to extend dbterd to support new diagramming tools or custom internal formats.

    Existing adapters include:

    • DBML: For dbdiagram.io
    • Mermaid: For Markdown embedding
    • PlantUML: Classic UML
    • GraphViz: DOT language
    • D2: Modern declarative diagrams
    • DrawDB: JSON format for DrawDB

  2. What is an Algorithm Adapter?

    main
    An algorithm adapter is a plugin for dbterd responsible for parsing dbt artifacts (manifest and catalog) to extract tables and relationships. While dbterd provides built-in algorithms like test_relationship, semantic, and model_contract, custom adapters allow you to implement unique relationship detection logic, such as using naming conventions (e.g., *_id columns), comments, or tags.

  3. How external adapter plugins work in dbterd

    main

    dbterd supports external adapter plugins via Python entry points. This allows you to develop custom algorithm or target adapters in separate Python packages (private or public) without forking the main dbterd repository.

    When dbterd starts, it scans the dbterd.adapters entry point group. It imports the modules specified in your package metadata, which triggers the @register_algo or @register_target decorators. These decorators then register your adapter in the PluginRegistry automatically.

    Your adapter does not need to be published to PyPI; it only needs to be installed in the same environment as dbterd using methods like local editable installs, Git URLs, or private registries.

    dbterd starts
    │
    ├── Scan built-in adapter modules
    │   └── import dbterd.adapters.algos.*
    │   └── import dbterd.adapters.targets.*
    │
    └── Scan entry points in "dbterd.adapters" group
        └── Load each entry point module
            └── @register_algo / @register_target decorators fire
                └── Adapter registered in PluginRegistry

  4. Accessing Manifest and Catalog data in Adapters

    main

    When implementing an adapter, you interact with two primary data structures: the Manifest and the Catalog.

    Manifest Structure

    The Manifest object contains dbt project definitions:

    • manifest.nodes: Dictionary of models, tests, seeds, and snapshots.
    • manifest.sources: Dictionary of source definitions.
    • manifest.exposures: Dictionary of exposures.
    • manifest.semantic_models: Dictionary of semantic models (dbt 1.6+).

    Node Properties:

    • node.name: The name of the node.
    • node.database: The database name.
    • node.schema_: The schema name (note the underscore).
    • node.columns: Dictionary of column metadata.
    • node.depends_on.nodes: List of upstream dependencies.
    • node.meta: Custom metadata dictionary.
    • node.description: Model description.

    Catalog Structure

    The Catalog object contains runtime database information:

    • catalog.nodes: Dictionary containing column types from the actual database.
    • catalog.sources: Source column information.

    Catalog Node Properties:

    • catalog_node.columns: Dictionary with actual column types (e.g., catalog_node.columns["id"].type returns "INTEGER").

  5. Choose an algorithm (parser) for Entity Relationship detection

    main

    When running dbterd, you can choose from three different algorithms (parsers) to detect relationships between dbt models. The choice depends on how you define your relationships in dbt:

    1. Test Relationship (Default): Uses dbt relationships data tests. This is the best choice if you already use dbt tests to ensure data consistency.
    2. Semantic Entities: Uses dbt Semantic Layer entities (primary and foreign). Use this if you have already implemented dbt Semantic Modeling (MetricFlow).
    3. Model Contract: Uses dbt Model Contracts (available in dbt v1.9+). This is the most direct method as it reads foreign_key and primary_key constraints directly from the model definitions without needing separate tests or semantic models.

  7. How entity relationship detection algorithms work

    main

    dbterd uses three different algorithms to detect relationships in your dbt project. You can select the algorithm that best matches how your project defines data contracts using the -a flag:

    1. Test Relationships (Default): Infers relationships from dbt relationships data tests.
    2. Semantic Entities (-a semantic): Detects relationships via dbt Semantic Layer entity definitions.
    3. Model Contract Constraints (-a model_contract): Detects relationships via dbt model contract foreign_key constraints (requires dbt 1.9+ and manifest v12+).

    You can also provide custom relationship detection logic using a specific string format via the -a flag.

  8. Supported configuration file formats and priority

    main

    dbterd automatically discovers configuration files in your current directory. It supports two formats:

    1. .dbterd.yml: YAML configuration (recommended).
    2. pyproject.toml: TOML configuration using the [tool.dbterd] section.

    Priority Order: If both files exist, dbterd follows this precedence:

    1. pyproject.toml (if it contains [tool.dbterd])
    2. .dbterd.yml

    Note: CLI arguments always override values found in configuration files.

  9. Include dbt lineage with `--with-dependencies`

    main

    The --with-dependencies (-wd) flag adds the dbt DAG (lineage) to your diagram as DBML [Dep] blocks. This is only supported for the dbml target.

    Key Concepts:

    • Relationships (Ref): Describe how entities join (detected via tests/contracts).
    • Dependencies (Dep): Describe how entities are built (the dbt DAG).
    • Selection Persistence: Lineage connections persist even if intermediate nodes are excluded via --select. The walk continues to the nearest selected ancestor.

    Examples:

    # Emit both relationships and DAG lineage
dbterd run --with-dependencies -t dbml

# Include sources in the lineage
dbterd run -wd -rt model -rt source -enf resource.package.table -t dbml
    dbterd run --with-dependencies -t dbml

  10. Use dbt selection logic with `--dbt`

    main

    By default, dbterd uses its own selection logic (e.g., prefix matching). If you want to use standard dbt selection syntax (like +model_name for upstream dependencies), use the --dbt flag.

    Note: To automatically generate the required artifacts for this to work, combine it with --dbt-auto-artifacts.

    Examples:

    # Select 'something' and all its upstream dependencies using dbt logic
dbterd run -s +something --dbt

# Select 'something' and its upstream, while forcing artifact generation
dbterd run -s +something --dbt --dbt-auto-artifacts

# Specify a specific dbt project and target when using dbt selection
dbterd run -s +something --dbt --dbt-project-dir /path/to/project --dbt-target prod
    dbterd run -s +something --dbt

  11. Use `dbterd run` to generate ERDs

    main

    The dbterd run command generates diagram-as-a-code files from dbt artifact files (like manifest.json and catalog.json). It can either read local artifacts from a specified directory or download them from the dbt Cloud Administrative API.

    Basic Workflow:

    1. Ensure dbt artifacts are generated (e.g., via dbt docs generate).
    2. Run dbterd run with your desired target format (e.g., dbml, mermaid, d2).

    Common CLI Examples:

    # Generate DBML ERD from a local artifacts directory
dbterd run --artifacts-dir ./target

# Generate Mermaid ERD from a local artifacts directory
dbterd run --artifacts-dir ./target --target mermaid

# Generate D2 ERD
dbterd run --artifacts-dir ./target --target d2

# Generate GraphViz ERD without columns (Mermaid only)
dbterd run --target graphviz --omit-columns
    dbterd run --artifacts-dir ./target --target mermaid

  12. Configure dbt Cloud environment variables

    main

    To use the run-metadata command, you must provide connection details via environment variables or CLI options.

    VariableCLI OptionDescription
    DBTERD_DBT_CLOUD_HOST_URL--dbt-cloud-host-urlThe Access URL with the metadata. prefix (e.g., metadata.cloud.getdbt.com).
    DBTERD_DBT_CLOUD_ENVIRONMENT_ID--dbt-cloud-environment-idThe dbt Cloud environment ID.
    DBTERD_DBT_CLOUD_SERVICE_TOKEN--dbt-cloud-service-tokenThe dbt Cloud Service Token with Metadata Only permissions.

    Setting variables in Linux/macOS

    export DBTERD_DBT_CLOUD_HOST_URL=metadata.cloud.getdbt.com
export DBTERD_DBT_CLOUD_SERVICE_TOKEN=your_token_here
export DBTERD_DBT_CLOUD_ENVIRONMENT_ID=your_env_id

    Setting variables in PowerShell

    $env:DBTERD_DBT_CLOUD_HOST_URL="metadata.cloud.getdbt.com"
$env:DBTERD_DBT_CLOUD_SERVICE_TOKEN="your_token_here"
$env:DBTERD_DBT_CLOUD_ENVIRONMENT_ID="your_env_id"