DOCSEARCHbyAlgolia
Submit Docs

Bauplan Documentation

website·Indexed 34 days ago

https://docs.bauplanlabs.com

A Python-first lakehouse runtime and 'Git-for-data' platform that allows developers to run Python and SQL pipelines with transactional version control. Bauplan provides features for data branching, commits, tags, and Apache Iceberg table integration, supporting both CLI and SDK workflows. It integrates with orchestrators such as Airflow, Temporal, and Dagster.

Tokens
45.1K
Snippets
215
Records
316
Agent score
65%

What's inside Bauplan

  1. Overview of Bauplan Git-for-data lakehouse

    Bauplan is a Git-for-data lakehouse designed to run Python and SQL pipelines directly on Apache Iceberg tables stored in the user's own object storage. It eliminates the need to manage servers or clusters and provides version control capabilities (branching, committing, and rolling back) for data pipelines.

  2. Overview of Bauplan Git-style version control for data lakes

    Bauplan implements Git-style version control for data lakes using Iceberg tables. This allows developers to manage data with transactional pipelines, commits, refs, tags, and zero-copy branches to ensure data changes can be safely imported, tested, and merged.

  3. Overview of Bauplan Lakehouse Runtime

    Bauplan is a Python-first lakehouse runtime designed for building and operating data pipelines on object storage (e.g., AWS S3). It implements Git-style branching and versioning for data, allowing developers and AI agents to write to isolated branches, validate changes, and then perform an atomic merge to the main branch. This model ensures that production data is protected and that changes can be rolled back to a known-good commit.

  4. Overview of Estuary and Bauplan integration

    The Estuary integration allows streaming data from any Estuary source into Bauplan Iceberg tables. It uses the Bauplan connector to orchestrate Spark jobs on AWS EMR Serverless, enabling near real-time merges of row-level changes (inserts, updates, deletes) via Change Data Capture (CDC). This decouples the ingestion pipeline from transformation logic within Bauplan.

  5. Overview of Bauplan Tables

    Bauplan uses Apache Iceberg tables to provide transactional, SQL-ready structure on object storage. This architecture combines the scalability of data lakes with the reliability of a data warehouse, supporting ACID transactions, schema evolution without downtime, efficient querying (via column pruning, partition filtering, and file skipping), and snapshot-based time travel. In Bauplan, tables are versioned and reside within branches.

  6. Overview of Bauplan Expectations

    Expectations are statistical and quality checks that validate the structure and values of tabular data (tables or dataframes). They function as boolean tests: returning True if data meets the standard and False if a compliance issue is detected.

    Key characteristics include:

    • Performance: Leverages vectorized SIMD operations to run at production speed on full tables or partitions.
    • Inline Execution: Evaluated during pipeline execution, acting as gates in a Write-Audit-Publish (WAP) pattern to block invalid data from being published.
    • Auditability: Results are recorded with the specific run, branch, and commit.

  7. Overview of Bauplan branch-based isolation

    Bauplan replaces traditional dev/staging/prod environments with branch-based isolation. Every branch operates on a full, isolated copy of the data catalog using Apache Iceberg, allowing multiple developers to work simultaneously without risking production data corruption. A fundamental principle of this system is that the only way to change data is by running code, ensuring every data state is reproducible and isolated until explicitly merged.

  8. Overview of Bauplan Git-style version control for data lakes

    Bauplan implements Git-style version control for data lakes using Iceberg tables. This system provides transactional pipelines, commits, refs, tags, and zero-copy branches, allowing users to manage data changes with the same rigor as source code.

  9. Overview of Bauplan standard expectations

    Bauplan provides a module of standard expectation functions to test data artifacts within a pipeline. Using these built-in expectations is recommended over custom implementations as they are more maintainable, faster, and more memory-efficient. Each expectation function returns a boolean value, allowing them to be used within assertion wrappers to trigger failures or log messages.

  10. Configure AWS EMR Serverless for Estuary

    Estuary requires specific AWS resources to run Spark jobs that merge data into Iceberg tables:

    • EMR Serverless Application: Use Spark runtime, release version emr-7.12.0, and architecture x86_64. Enable 'Automatically start application on job submission'. Ensure the VPC/security groups allow access to the Bauplan catalog endpoint: https://api.use1.aprod.bauplanlabs.com/iceberg.
    • Staging S3 Bucket: Create a bucket in the same region as the EMR application. A dedicated prefix (e.g., estuary-staging/) within the Bauplan data lake bucket is recommended for staging Parquet/CSV files and Python scripts.

  11. Reference the `city_bike_nyc` dataset schema

    The city_bike_nyc dataset contains information about Citi Bikers in NYC. It consists of 366,676,951 rows. Key columns include station_id (string), num_bikes_available (int), num_docks_available (int), station_name (string), and coordinates lat and lon (string).

  12. Integrate Bauplan tables with BigQuery via Iceberg

    Bauplan allows you to surface managed data in BigQuery without copying it by creating external Iceberg tables in BigQuery that point to Bauplan's S3 storage. This setup uses a BigQuery cross-cloud connection to read objects from S3 and the Iceberg table format to discover metadata. This is ideal for users who want BigQuery users to run SQL over Bauplan tables or prefer read-only access from BigQuery while maintaining data evolution in Bauplan.