Bruin Documentation

The Insights dropdown in the top navigation provides access to four distinct analytics and observability views. The availability of these views depends on your Bruin Cloud plan and specific permissions.

The Spark integration test suite provides comprehensive coverage for the following Bruin features:

• Connectivity & Queries: Connection testing and ad hoc queries.
• Materializations: create/replace, append, delete/insert, truncate/insert, merge, time-interval, DDL, SCD2-by-column, SCD2-by-time, and view materializations.
• Iceberg Features: Iceberg partition specs and sort-order-backed clustering for create/replace, DDL, and SCD2 tables.
• Quality & Assets: Annotated Spark asset and quality-check queries, including successful/failing column checks and custom checks.
• Ingestion & Catalog: CSV seed ingestion via ADBC's S3 staging path, automatic catalog namespace creation, and database import through ADBC GetObjects (including columns).
• Sensors: Query sensors and metadata-based table sensors.
• Assets: Source assets.

The oracle-duckdb template demonstrates a pattern of ingesting operational data from Oracle into DuckDB and then performing transformations.

Ingestion Assets (Oracle to DuckDB via ingestr):

• oracle_raw.customers: Customer master data.
• oracle_raw.orders: Main orders fact table.
• oracle_raw.order_items: Order line items.

Transformation Asset (DuckDB):

• duckdb.sales_per_customer: An aggregated table that joins the three staging tables to provide order metrics.

The NYC Taxi Pipelines project is a comprehensive ELT (Extract, Load, Transform) template built with Bruin. It demonstrates production-ready data pipeline best practices by processing NYC taxi trip data from public HTTP sources through a multi-tier architecture:

1. Raw Layer: Ingestion and raw data storage.
2. Staging Layer: Cleaned and deduplicated data.
3. Reports Layer: Aggregated analytical reports.

This project showcases core Bruin features such as Python asset materialization, incremental processing strategies, pipeline variables, and data quality checks.

Bruin is a data pipeline framework that integrates data ingestion, transformation (SQL, Python, and R), and data quality. It is designed to run on local machines, EC2 instances, or GitHub Actions and supports major data platforms.

Key capabilities include:

• Ingestion: Use ingestr or Python for data ingestion.
• Transformations: Execute SQL, Python, and R transformations across various platforms.
• Materializations: Manage table/view materializations and incremental tables.
• Python Isolation: Run Python code in isolated environments using uv.
• Data Quality: Built-in data quality checks.
• Templating: Use Jinja templating to reduce repetition.
• Validation: Validate end-to-end pipelines using dry-run.
• Security: Inject secrets via environment variables.
• Developer Experience: Supports a VS Code extension and is written in Golang.

The import command allows you to automatically import existing resources as Bruin assets. This automates the creation of asset definition files for various source types, facilitating the migration of existing data infrastructure into a Bruin pipeline.

Available Subcommands:

• bruin import database: Import database tables as Bruin assets.
• bruin import bq-scheduled-queries: Import BigQuery scheduled queries as Bruin assets.
• bruin import odi: Import Oracle Data Integrator XML exports as Bruin assets.
• bruin import tableau: Import Tableau dashboards, workbooks, and data sources as Bruin assets.
• bruin import quicksight: Import QuickSight datasets and dashboards as Bruin assets.

This template demonstrates a canonical ELT pattern using two primary assets:

• Bronze Asset (assets/bronze_raw_data.asset.yml): Uses ingestr to copy foreign exchange rates from the Frankfurter API into PostgreSQL. It includes data quality features such as column-level checks, deduplication assurance, and freshness validation.
• Silver Asset (assets/silver_aggregated.sql): A SQL-based transformation that builds a summarized silver table in PostgreSQL. It provides exchange rate insights, rolling averages, and observation counts for downstream analytics.

This template demonstrates an end-to-end analytics pipeline using a multi-layer architecture:

• Bronze Layer: Python assets (assets/bronze/*.py) generate deterministic source-shaped data for products, retailers, stores, sales, distribution, inventory, promotion, and cost.
• Silver Layer: Snowflake SQL assets (assets/silver/*.sql) normalize the bronze data into fact tables (e.g., sku_daily_sales) and analytical marts (e.g., limited_edition_performance, retailer_channel_scorecard).
• Gold Layer: Snowflake SQL assets (assets/gold/*.sql) produce decision-ready outputs such as lifecycle recommendations, decision drivers, channel alerts, and weekly SKU council summaries.

The Athena template consists of a pipeline.yml configuration file and several SQL assets located in the assets/ folder. The assets include:

• cars.sql: Defines an Athena cars table (schema, constraints, and sample data).
• drivers.sql: Creates an Athena drivers table (schema, constraints, and data).
• payments.sql: Defines an Athena view for payments with positive amount checks.
• travellers.sql: Creates an Athena travellers table (schema, constraints, and sample data).

This pipeline follows a bronze-to-silver ELT pattern, moving data from Stripe into Databricks.

Bronze Layer (Raw Ingestion)

These assets use ingestr to collect raw data from the Stripe API:

• bronze_customer_data_raw: Customer data
• bronze_subscription_data_raw: Subscription data
• bronze_charge_data_raw: Charge/payment data
• bronze_balance_transaction_data_raw: Balance transaction data

Silver Layer (Transformed)

This layer provides curated, joined data:

• silver_customer_subscription_simple: Joins customers with their subscriptions and their most recent balance transaction.

Data Model Logic

Because Stripe balance transactions do not have a direct customer field, the silver layer joins them through the charge entity using this relationship: customer ← charge (via charge.customer) → balance_transaction (via charge.balance_transaction)