DOCSEARCHbyAlgolia
Submit Docs

dbt Core

repository·main·Indexed 11 days ago

https://github.com/dbt-labs/dbt-core

A development framework for data engineers and analysts to transform data in warehouses using software engineering best practices. v2.0 is a high-performance Rust rewrite. The project includes dbt-adbc for Arrow Database Connectivity drivers, dbt-ci for release pipeline automation (PyPI and Homebrew), and dbt-cloud-api for interacting with the dbt Cloud API v3.

Tokens
168.6K
Snippets
507
Records
844
Agent score
96%

What's inside dbt

  1. Overview of MiniJinja features and goals

    main

    MiniJinja is a minimal-dependency template engine for Rust based on the Jinja2 (Python) syntax. It is built on top of serde and aims to provide a powerful yet compact API for tasks like HTML generation, structure generation, and LLM chat templating.

    Key Features:

    • Jinja2 Compatibility: Supports inheritance, filters, and expression evaluation.
    • Minimal Dependencies: Only requires serde.
    • Data Support: Works with all serde compatible types and supports dynamic runtime objects via the Object trait.
    • Portability: Compiles to WebAssembly and has experimental C-bindings.
    • Extensibility: Available via various ecosystem crates (e.g., minijinja-py for Python, minijinja-cli for CLI).

  2. Overview of dbt-tracing

    main

    dbt-tracing is a structured telemetry layer built on top of the tracing crate. It provides a generic framework for handling typed span and log attributes, telemetry record envelopes, middleware, consumers, filtering, and serialization/export.

    Note: This is not a product analytics client or anonymous usage telemetry. It is a library that allows applications to define their own event taxonomy, Arrow schemas, and registry lookup behavior for dynamic attributes.

  3. Overview of dbt Core v2.0 (Alpha)

    main

    dbt Core v2.0 is a major rewrite of the framework powered by a Rust engine. It aims to provide a single, high-performance foundation for both dbt Core and Fusion distributions.

    Key features include:

    • Performance: Built with Rust and designed for scale (utilizing technologies like ADBC).
    • Strict Specification: A codified language spec that eliminates silent failures from misspelled or misplaced configurations.
    • New Artifacts: New Parquet metadata artifacts designed for speed and scale, which power a redesigned dbt-docs experience.
    • Unified Foundation: A single engine and adapter layer shared across distributions.

  4. Overview of dbt-cloud-api models

    main

    The dbt-cloud-api crate provides a comprehensive set of Rust models that map to the dbt Cloud API. These models cover various domains including:

    • Account & User Management: AccountUser, AccountResponse, User, AccountListEnveloped.
    • Connections & Adapters: BigqueryConnection, SnowflakeConnection, PostgresConnection, AdapterMetadata, ConnectionTypeEnum.
    • Project & Environment: Project, EnvironmentV3, RepositoryV3.
    • Security & Access: ServiceToken, ScimConfig, IpRestrictionRule, GroupPermission.
    • Jobs & Automation: JobDefinitionV2, JobTypeEnum.
    • Webhooks & Audit: WebhookResponseStatus, AuditLogBulkExportStatusResponse.

    Detailed documentation for each specific model is available in the generated docs/ directory of the crate.

  6. Upcoming dbt Core features and roadmap

    main

    The dbt Core roadmap includes several key architectural and functional improvements:

    Adapter Ergonomics

    • Grants: Moving away from relying on pre-hook and post-hook for database permissions to provide more consistent and easier-to-use grants functionality.
    • Materializations: Improving the incremental materialization and incremental strategy logic to be more sensible and easier to customize/contribute to.
    • dbt-utils splitting: Moving low-level "cross-db" macros from dbt-utils into dbt-core and plugins to expand the dbt-SQL language.

    Modular Interfaces & Metadata

    • Decoupled CLI: Unbundling the CLI from the dbt-core library to support better programmatic interfaces for initialization and tasks.
    • Structured Logging: Moving toward a real-time event system for metadata (e.g., model table statistics during dbt run) that may eventually supplement or eclipse manifest.json.

    Unified Lineage & Future Work

    • Python Models: Production-ready support for Python-language models.
    • UDF Support: Native support for User Defined Functions.
    • Cross-project Lineage: Support for namespaced models to facilitate large-scale multi-project environments.
    • SQL Grammar: Improved parsing, linting, and column-level lineage detection.

  7. Use dbt-tui-progress for terminal progress bars

    main

    The dbt-tui-progress crate provides a thread-safe, type-safe API for managing terminal progress bars and spinners in TUI (Terminal User Interface) layers. It wraps the indicatif crate and allows you to identify progress bars using any hashable type (e.g., an enum), decoupling the identity of a task from its display text.

    Key features include:

    • Generic ID type: Use custom enums or types to track progress.
    • Thread-safety: Uses scc::HashMap for concurrent access.
    • Background animations: A dedicated ticker thread handles animations.
    • Suspension support: Provides a mechanism to interleave log output with progress bars without corrupting the terminal display.
    use dbt_tui_progress::ProgressController;

#[derive(Debug, Clone, Hash, Eq, PartialEq)]
enum Phase {
    Render,
    Run,
}

let mut ctrl = ProgressController::<Phase>::new();
ctrl.start_ticker();

// Start a progress bar
ctrl.start_bar(Phase::Render, 100, "Rendering");

// Track in-progress items
ctrl.add_bar_context(&Phase::Render, "model_a");
ctrl.finish_bar_context(&Phase::Render, "model_a", Some("succeeded"));

// Suspend for log output
ctrl.with_suspended(|| {
    println!("Log message");
});

// Clean up
ctrl.remove_bar(&Phase::Render);

  9. Authenticate to the dbt platform using `dbt-platform-auth`

    main

    The dbt-platform-auth crate provides a mechanism to resolve credentials (Service Tokens, Personal Access Tokens, or OAuth sessions) required to authenticate requests to the dbt platform. It uses an AuthChain which is an ordered sequence of resolvers that are tried in order until one successfully provides a Credential.

    use dbt_platform_auth::AuthChainBuilder;

let chain = AuthChainBuilder::default().build();
let credential = chain.resolve().await?;

println!("host:  {}", credential.account_host());
println!("token: {}", credential.token());

  10. Historical dbt Core version features and milestones

    main

    A summary of key features introduced in recent dbt Core minor versions:

    • v1.5: Introduced a revamped CLI, programmatic invocations (invoking dbt as a Python module), and model governance features including contracts, access, groups, and versions.
    • v1.6: Introduced the new Semantic Layer specification, expanded model governance (deprecations), retry + clone functionality to save time and cost, and the initial rollout of materialized views.
    • v1.7: Improved access to 'applied state' in docs generate and source freshness, and further enhancements to model governance and Semantic Layer features.
    • v1.8 (Planned): Focuses on stability, including stable interfaces for adapters and artifacts, and built-in support for unit testing dbt-SQL models.

  11. Embed MiniJinja templates into a binary

    main

    MiniJinja-Embed is a utility crate for MiniJinja designed to facilitate embedding templates directly into a Rust binary. It provides two primary workflows:

    1. Iterative Development: Supports using a loader to fetch templates from the file system, allowing for rapid changes without recompiling.
    2. Production Deployment: Provides utility macros to embed the templates directly into the compiled binary for single-file distribution.

    Requires Rust 1.63.0 or higher.

  12. dbt Core Roadmap (August 2022)

    main

    This roadmap outlines the planned evolution of dbt Core, detailing version releases, key features, and development confidence levels.

    Release Schedule and Key Features

    VersionExpected ReleaseKey Features
    v1.1AprilTesting framework for dbt-core + adapters; sustainable OSS maintenance tools.
    v1.2JulyBuilt-in support for grants; migration of cross-db macros into dbt-core/adapters; metrics improvements.
    v1.3OctoberPython models in dbt; improvements to metrics; support for custom node colors; Jinja3 upgrade.
    v1.4Jan 2023Technical interface improvements; documented Python API/library; improved CLI; structured logging.
    v1.5+Next YearMulti-project deployments (splitting the monolith); external orchestration; next steps for Python in dbt; dbt Core v2 planning.