DOCSEARCHbyAlgolia
Submit Docs

SDP-META (Spark Declarative Pipelines)

repository·main·Indexed 42 days ago

https://github.com/databrickslabs/dlt-meta

A metadata-driven framework for building Lakeflow Spark Declarative Pipelines. It enables the definition of Bronze and Silver medallion architecture pipelines using JSON/YAML configuration instead of custom Spark code. Formerly known as dlt-meta, the framework includes a Databricks App for deployment and management.

Tokens
81.3K
Snippets
215
Records
351
Agent score
53%

What's inside SDP-META

  1. What is SDP-META?

    main

    SDP-META is a metadata-driven framework designed to automate the construction of Bronze and Silver data pipelines on Databricks Lakeflow Spark Declarative Pipelines.

    Instead of writing unique pipeline code for every table, you define your data requirements in a JSON or YAML onboarding file. SDP-META uses a single generic Declarative Pipeline that reads this metadata at runtime to dynamically build the full processing graph, including sources, transformations, data quality rules, CDC behavior, and sinks.

  2. Overview of SDP-META

    main

    SDP-META is a metadata-driven framework designed for Lakeflow Spark Declarative Pipelines. It allows platform and data engineering teams to define Bronze and Silver pipelines using JSON or YAML onboarding files. A single generic Declarative Pipeline reads the resulting DataflowSpec at runtime to automatically build the full processing graph, eliminating the need to write custom pipeline code for every new dataset.

    Key Use Cases:

    • Standardizing repeatable Bronze/Silver pipelines across many datasets.
    • Onboarding new feeds via metadata instead of new code.
    • Implementing consistent data quality, quarantine, CDC, clustering, and sink patterns.

    When NOT to use SDP-META:

    • For one or two simple pipelines.
    • For Gold-layer business modeling.
    • For tables requiring unique, complex application logic.
    • When a formal support SLA is required (this is a Databricks Labs project).

  3. Available usage guides in DLT-META

    main

    The DLT-META project provides specialized guides for common data engineering patterns using Delta Live Tables (DLT). Use these guides to implement specific ingestion and transformation strategies:

    • Cloud Storage Ingestion: Use the Autoloader / Cloud Files guide to ingest files landing in S3, ADLS, or GCS using Databricks Autoloader.
    • Streaming Ingestion: Use the Kafka & Event Hubs guide to ingest data from Kafka or Azure Event Hubs.
    • Change Data Capture (CDC): Use the CDC with apply_changes guide to implement Type 1 or Type 2 SCD from a CDC source.
    • Snapshot Ingestion: Use the Snapshot Ingestion guide for full snapshot ingestion with replace-all semantics (non-streaming).
    • Data Fan-out: Use the Silver Fanout guide to use one bronze table as a source for multiple silver tables.
    • External Sinks: Use the DLT Sink guide to write pipeline output to an external Delta table or Kafka topic.
    • Multi-Source Aggregation: Use the Multi-Source CDC guide to merge multiple source paths (e.g., US, EU, APAC) into a single target table.
    • Data Filtering: Use the Row Filters guide to filter rows at pipeline time based on column values.

  4. Choose a deployment path for SDP-META

    main

    SDP-META offers several ways to onboard metadata and run Bronze/Silver pipelines depending on your workflow requirements:

    PathBest ForDescription
    Declarative Automation Bundles (DABs)Production & TeamsGit-tracked configuration with explicit dev/prod environments. CI/CD friendly.
    Interactive CLIExploration & One-offsOnboard and deploy using interactive prompts without needing bundle YAML files.
    Databricks AppNon-EngineersA browser-based GUI for managing pipelines without using a terminal.
    MCP ServerAI-Assisted ScaffoldingUse natural language via Claude Code, Claude Desktop, or Cursor to drive configuration.
    Agent SkillAI Agent IntegrationA portable skill that teaches AI agents the SDP-META workflow to guide you from raw data to a running pipeline.

  5. Reference the SDP-META technical documentation

    main

    The SDP-META technical reference documentation is organized into several key areas to help you configure onboarding, transformations, data quality, and CLI operations:

    • Onboarding File Fields: Detailed reference for the onboarding JSON/YAML file, covering top-level fields, source_details (per-format), bronze layer fields, and silver layer fields.
    • Silver Transformations Schema: Schema reference for the silver transformations file, including field definitions and JSON/YAML examples.
    • Data Quality Rules Schema: Schema reference for DQE (Data Quality Engine) rules files, specifically defining expect, expect_or_drop, and expect_or_fail constraint types.
    • CLI Commands: Full reference for all databricks labs sdp-meta commands, flags, and usage examples.
    • DAB Bundle Parameters: Reference for Databricks Asset Bundle (DAB) scaffold prompts and resources/variables.yml keys.

  6. SDP-META Feature Capabilities Overview

    main

    The SDP-META project provides several advanced data engineering capabilities demonstrated in its demos:

    • Metadata-driven Onboarding: Convert JSON or YAML specifications into DataflowSpec tables to automate pipeline creation.
    • Automated CDC: Use dp.create_auto_cdc_flow to merge multiple regional CDC sources into a single silver target table using select_exp for normalization.
    • Append Flow: Use dp.append_flow to ingest from multiple sources into the same target table, including file metadata columns (_metadata.file_name, _metadata.file_path).
    • Snapshot Ingestion: Implement SCD Type 1 & 2 using apply_changes_from_snapshot from CSV or Delta snapshots.
    • Data Quality & Quarantine: Implement expect_or_drop and expect_or_quarantine rules with dedicated quarantine tables.
    • Row-Level Security: Implement UC Row-Level Security via bronze_row_filter and silver_row_filter UDFs.
    • External Sinks: Use dp.create_sink to write Bronze outputs to external Delta destinations.

  7. What is the SDP-META Agent Skill

    main

    The SDP-META Agent Skill is a portable, agent-facing operating manual written in the Agent Skills format. It consists of a SKILL.md entry point and a references/ directory containing technical specifications.

    Its purpose is to teach AI coding agents how to use SDP-META correctly by providing knowledge on:

    • The onboarding metadata model.
    • The onboard → deploy workflow.
    • Databricks Asset Bundle (DAB) commands.
    • Common pitfalls and guardrails.

    While the MCP Server provides the tools the agent calls to execute actions (scaffolding, validating), the Agent Skill provides the knowledge that steers the agent to perform those actions in the correct order and with the correct parameters.

  8. How SDP-META App boot modes work

    main

    The databricks_app/start.sh script uses two different modes depending on the source-code path provided during deployment:

    • Mode A (CLI Deploy): The App's source-code path is the repository root. start.sh builds the wheel and starts Flask directly from the local files. This is used when deploying via scripts/deploy_app.sh or scripts/deploy_app.ps1.
    • Mode B (Git Folder UI): The App's source-code path is set to databricks_app/. Because src/ and demo/ are not siblings to this folder in the container, start.sh detects this state and clones the full databricks/sdp-meta repository into /tmp/sdp-meta at container start. All builds and execution run from this fresh clone.

  9. Define Data Quality (DQ) expectations

    main

    Data quality expectations are defined in a JSON object keyed by an action. Each action maps a constraint name to a SQL predicate.

    Actions and Behaviors:

    • expect: Record violations; keep the row (warn).
    • expect_or_drop: Drop violating rows (routed to the quarantine table if configured).
    • expect_or_fail: Fail the pipeline update on any violation.

    Example DQ file:

    {
  "expect_or_drop": {
    "no_rescued_data": "_rescued_data IS NULL",
    "valid_id": "id IS NOT NULL",
    "valid_operation": "operation IN ('APPEND','DELETE','UPDATE')"
  }
}
    {
  "expect_or_drop": {
    "no_rescued_data": "_rescued_data IS NULL",
    "valid_id": "id IS NOT NULL",
    "valid_operation": "operation IN ('APPEND','DELETE','UPDATE')"
  }
}

  10. How Dataflowspec works in SDP-META

    main

    A Dataflowspec is a configuration file (YAML or JSON) used to define data pipelines. The system operates in two distinct phases:

    1. Onboard Job: This job reads your onboarding file and writes the metadata into two Delta tables: bronze_dataflowspec and silver_dataflowspec. You must re-run this job whenever you modify your onboarding file.
    2. Generic Declarative Pipeline: At startup, this pipeline reads the bronze_dataflowspec and silver_dataflowspec tables to dynamically construct the pipeline graph.

    Important: The Delta tables are derived artifacts. Always edit the onboarding file and re-run the Onboard Job. Do not edit the rows in the bronze_dataflowspec or silver_dataflowspec tables directly, as they will be overwritten during the next run.

    # Example of a single flow entry in an onboarding file
- data_flow_id: '1'
  data_flow_group: retail
  source_format: cloudFiles
  # ... other fields

  11. Use the SDP-META Databricks App

    main

    The SDP-META Databricks App provides a browser-based UI for onboarding, deploying, and monitoring pipelines. It is designed for non-developers, demos, and human-in-the-loop workflows.

    Note: The App is for exploration and demos. For production CI/CD, you should graduate to the Declarative Automation Bundle (Path A).

  12. Configure the Filesystem Sandbox via SDP_META_MCP_ROOT

    main

    To prevent unauthorized file access, the MCP server confines all caller-supplied paths (output_dir, bundle_dir, config_file, onboarding_file) to a single root directory.

    You must set the SDP_META_MCP_ROOT environment variable to identify this root. If this variable is absent, bundle initialization, validation, and flow operations will refuse to run. Paths attempting to escape this root are rejected before any write operations occur.