dbt Documentation

Moving from self-hosted dbt to the dbt platform transitions your workflow from a manual, command-line-only setup to a fully managed service. The dbt platform provides managed development experiences (Studio IDE or CLI), out-of-the-box CI/CD workflows, the dbt Semantic Layer, multi-project Mesh setups, and a Catalog for data discovery.

Key differences:

• dbt (Open Source): Requires manual maintenance of production deployments, self-hosted orchestration, and command-line-based development.
• dbt platform: Provides a managed service for development, testing, deployment, and exploration, reducing the total cost of ownership and improving governance.

The dbt Semantic Layer is a dbt offering that allows you to centrally define metrics within your dbt project using MetricFlow. It provides a consistent way to manage metric logic and serves it to downstream tools.

Key Capabilities:

• Dynamic SQL generation: Automatically computes metrics based on your definitions.
• APIs: Provides interfaces to query metrics and dimensions.
• Integrations: Supports first-class integrations to query centralized metrics in various downstream BI and data tools.

Core Engine: The Semantic Layer is powered by MetricFlow, which is a source-available component.

dbt Charts is a visualization layer that converts YAML files into interactive dashboards. You define queries, charts, and layouts as text and store them in Git alongside your dbt project.

Key features:

• Direct Integration: Queries dbt models directly, ensuring dashboards stay in sync with your transformations.
• Local Development: Use the dct command-line tool to run queries against your existing dbt profiles.yml connection and preview results in a browser with hot reload.
• Version Control: All dashboard definitions (queries, charts, layouts) are stored as text files in your repository.

The dbt MCP (Model Context Protocol) server is an experimental tool that allows LLMs and AI agents to interact with your dbt project. It acts as a bridge between your dbt project (models, docs, lineage, Semantic Layer) and MCP-enabled clients like Claude Desktop, Cursor, or custom agent frameworks.

It provides three primary functional pillars:

1. Data Discovery: Understanding what data assets exist and how they relate.
2. Data Querying: Accessing data via the dbt Semantic Layer (for governed metrics) or via SQL execution (for exploration).
3. Project Execution: Running dbt CLI commands within the project.

Note: The server supports both dbt Core and dbt Cloud users. When using this in production, exercise caution with permissions; it is recommended to start in a sandbox environment or grant read-only permissions.

The dbt Cloud IDE received several updates in April 2023:

New Features:

• dbt deps warnings: The IDE now suggests invoking dbt deps when required (via dbt-score).
• Unsaved changes warning: A warning appears if you select models but do not save them before clicking Build or running dbt commands.
• File previews: Markdown and CSV file previews are available in the IDE console.
• Duplicate File: A new option in the file tree menu.
• Loading indicators: Displayed when previewing a model.

Product Refinements:

• Autocomplete performance: Improved for large projects by implementing a manifest.json limit.
• Pagination: The invocation node summary view now displays 100 nodes at a time.
• Lineage: The lineage tab has been restored for seed resources.

dbt Core v2.0 introduces a shared foundation between dbt Core and dbt Fusion, utilizing a high-performance Rust implementation. Key improvements include:

• Performance: Significant parse time improvements, especially for large projects.
• Language Specification: A tightly-defined language spec to prevent configuration errors (e.g., typos in description).
• New Artifacts: High-performance Parquet artifacts provided as an alternative to large JSON files, compatible with DuckDB or AI agents.
• Documentation: A revamped local documentation experience that scales to arbitrary project sizes.
• Adapter Development: A streamlined way to build new adapters using ADBC and the Arrow ecosystem.
• Installation: A simplified installation process that reduces reliance on Python virtual environments.

dbt Wizard is an AI agent that operates using your project's dbt metadata—including models, lineage, tests, and metrics—rather than relying solely on code analysis. This allows for more accurate context when performing tasks.

You can interact with Wizard in two ways:

1. Locally via the CLI: Requires the Wizard CLI to be installed and connected to a dbt project with an up-to-date target/manifest.json.
2. Via the dbt platform: Integrated directly into the dbt cloud/platform environment.

Recommended workflows include mapping unfamiliar codebases, validating changes, adding tests, debugging failed jobs, developing against production state, building Semantic Layer definitions, and extending Wizard with plugins.

The dbt system command is used to manage dbt v2 installations, which are single compiled binaries. It allows you to update, uninstall, or manage drivers directly without relying on package managers like pip, brew, or winget.

Important Compatibility Note: dbt system update and dbt system uninstall are only supported for CDN installations of dbt. If you installed dbt via Pip, Homebrew, or Winget, you must use those specific tools to update or uninstall it. dbt core does not support these commands.

dbt system <subcommand> [flags]

Supported Features

• Commands: dbt clean, dbt compile, dbt debug, dbt seed, and dbt run.
• Materializations: table and incremental.
• Apache Hudi table types: Merge on Read (MoR) and Copy on Write (CoW). Note: MoR is recommended for mutable workloads.

Limitations

• Views: Not supported.
• dbt seed: Has row/record limits and only supports Copy on Write (CoW) tables.

The animation directory is organized as follows:

• dag_model.py: Contains shared data and mobject factories (the demo DAG, color palette, multi-environment scene geometry, and the CLI line-color map). This file does not contain a Scene and is imported by other scene files.
• clone_demos.py: Contains the four animated clone demos.
• dbt_state_normalisation.py: Contains the query-normalization still and the volatile-SQL animation.
• cli_captures/*.txt: Contains curated terminal output from real dbt runs, which are replayed by the animations.
• specs/: Contains design and implementation notes for the scenes.