repository·main·Indexed 20 days ago
https://github.com/dbt-labs/dbt-fusionA high-performance, Rust-based rewrite of the dbt execution engine featuring enhanced SQL comprehension and faster data transfers via ADBC drivers. The repository includes the dbt-ci crate for release pipeline management (PyPI and Homebrew), the dbt-cloud-api crate for interacting with dbt Cloud API v3, and a custom tracing infrastructure for performance visualization via Jaeger and OTLP.
dbt-csv crate is a CSV reader designed to be 100% conformant with dbt Core's CSV parsing behavior. While dbt Core uses Python's agate library to parse CSVs into Python-typed values, dbt-csv replicates this logic in Rust. It produces a stream of Arrow RecordBatch objects with types that match what dbt Core would produce, ensuring consistent behavior across the dbt ecosystem.dbt Core v2.0 is a major rewrite powered by the Rust engine (previously known as Fusion). It aims to provide a single, unified foundation for the dbt framework.
Key features include:
dbt-docs experience.MiniJinja-Embed is a utility crate for the MiniJinja template engine. It provides two primary capabilities for developers:
This document outlines the strategic development goals for dbt Core throughout 2022. The roadmap focuses on three primary pillars:
New minor versions of dbt Core and its official adapters are released every three months:
For each minor version, a migration guide is provided in the official documentation.
The dbt-cloud-api crate provides a comprehensive set of Rust models that map to the dbt Cloud API. These models cover various domains including:
AccountUser, AccountResponse, User, AccountScopedPatResponse.BigqueryConnection, SnowflakeConnection, PostgresConnection, DbtAdapter, AdapterMetadata.Project, EnvironmentV3, RepositoryV3.ServiceToken, ScimConfig, IpRestrictionRule, GroupPermission.JobDefinitionV2, JobTypeEnum.WebhookResponseStatus, AuditLogBulkExportStatusResponse.Detailed documentation for each specific model is available in the docs/ directory of the crate.
The dbt_macro_assets component relies on adapter macros that are maintained in external repositories rather than within this specific crate. If you need to find or contribute to the macros used for specific adapters, refer to the following locations:
The dbt-ci crate provides commands for managing the release pipeline of dbt-fusion. These commands are exposed via a cargo ci alias in the workspace configuration. They handle version bumping, PyPI packaging/publishing, and Homebrew formula rendering/publishing.
# Commands are invoked via the cargo ci alias
cargo ci <subcommand> [options]dbt Core follows a release cycle of approximately one minor version per quarter. The roadmap includes planned features with varying confidence levels. Key historical and upcoming milestones include:
dbt-test-containers directory provides Dockerfiles designed to run tests in isolated environments. These images are specifically used by tests such as test_dbt_compile, which utilizes dbt/Dockerfile to execute dbt commands within an isolated Python environment. This ensures that test execution does not interfere with the host system's dependencies.The dbt-tui-progress crate provides a thread-safe, type-safe terminal progress bar controller for TUI (Terminal User Interface) layers. It wraps indicatif to manage multiple progress bars and spinners using a generic, hashable ID type. This allows you to decouple the identity of a task from its display text.
Key features include:
Hash + Eq.scc::HashMap for concurrent access.use dbt_tui_progress::ProgressController;
#[derive(Debug, Clone, Hash, Eq, PartialEq)]
enum Phase {
Render,
Run,
}
let mut ctrl = ProgressController::<Phase>::new();
ctrl.start_ticker();
// Start a progress bar
ctrl.start_bar(Phase::Render, 100, "Rendering");
// Track in-progress items
ctrl.add_bar_context(&Phase::Render, "model_a");
ctrl.finish_bar_context(&Phase::Render, "model_a", Some("succeeded"));
// Suspend for log output
ctrl.with_suspended(|| {
println!("Log message");
});
// Clean up
ctrl.remove_bar(&Phase::Render);Fusion uses a custom tracing architecture built on top of the native tracing crate to overcome limitations like lack of thread-safe event storage and restricted access to structured data during filtering.
The data flows through four main stages:
tracing macros (e.g., instrument, create_info_span!).tracing Layer that converts spans/events into structured records, generates unique IDs, and injects code location/context.TelemetryMiddleware to transform, modify, or drop spans/logs. It has mutable access to metrics via DataProviderMut.TelemetryConsumer to process data in a read-only fashion (e.g., writing to JSONL, Parquet, or exporting via OTLP).