DOCSEARCHbyAlgolia
Submit Docs

DQX (Data Quality eXperience)

repository·main·Indexed 19 days ago

https://github.com/databrickslabs/dqx

A Databricks Labs library for performing scalable data quality checks on PySpark workloads, compatible with batch and streaming data. It includes DQX Studio, a web application for authoring and managing quality rules, and supports integration with dbt core and dbt-databricks. The framework utilizes a distributed architecture with a FastAPI backend, React frontend, and serverless Databricks Jobs for executing profiling and validation tasks.

Tokens
178.4K
Snippets
359
Records
568
Agent score
65%

What's inside databricks-labs-dqx

  1. Overview of DQX Studio

    main
    DQX Studio is the web application interface for the DQX framework. It provides a user interface for authoring and managing data quality rules. The application is built using a FastAPI backend and a React frontend, and it is designed to be deployed as a Databricks App.

  2. Overview of the DQX MCP Server

    main
    The DQX MCP Server is an implementation of the Model Context Protocol (MCP) that exposes DQX data quality tools to AI agents. It is designed to run as a Databricks App and utilizes on-behalf-of (OBO) authentication to ensure that the AI agent operates with the appropriate security context.

  3. What is DQX Studio (No-Code UI)

    main

    DQX Studio is a web application deployed as a Databricks App within your workspace. It provides a no-code interface for data stewards and analysts to author, review, run, and monitor data quality rules.

    It is the recommended entry point for users who:

    • Want to manage quality rules without code-level integration.
    • Require an approval workflow before rules are activated.
    • Prefer browsing Unity Catalog and authoring checks through a graphical user interface.

    For programmatic integration into pipelines or notebooks, use the Programmatic approach instead.

  5. What is DQX Studio?

    main

    DQX Studio is a workspace-native UI and control plane built on top of the open-source databrickslabs/dqx data-quality engine. It is deployed as a Databricks App within a customer's own workspace, allowing for the authoring, approving, scheduling, and reviewing of data-quality rules using the customer's own compute and storage.

    Key differences from running DQX in a notebook:

    • Rule authoring UI: Form-based and YAML interfaces instead of notebook code.
    • Approval workflow: Separation of duties between RULE_AUTHOR and RULE_APPROVER.
    • Schedules: Automated runs via serverless jobs without manual notebook orchestration.
    • Run history: Centralized view of status, metrics, sample bad rows, and review status.
    • Insights dashboard: Embedded Lakeview/AI-BI dashboards.
    • Hybrid storage: Uses Lakebase (Postgres) for fast OLTP metadata and Delta for high-volume run history.

  6. Choose a DQX installation method

    main

    DQX can be installed in three distinct ways depending on your use case:

    1. Standalone Library: For code-level integration into Databricks pipelines, clusters, or notebooks. Use this for programmatic control.
    2. Workspace Tool: Deployed via the Databricks CLI. This method deploys configuration-driven profiling, quality-checkers, and end-to-end workflows.
    3. DQX Studio: A web UI deployed as a Databricks App. This is the recommended option for a no-code experience, allowing you to author, review, run, and monitor quality rules through a browser without maintaining configuration files.

  7. Approaches to Profiling and Quality Checks Generation

    main

    DQX provides three main ways to profile data and generate quality rule candidates:

    1. DQX Studio (No-code UI): The recommended option for a browser-based experience. It allows you to profile tables and review/save rule candidates without writing code.
    2. Programmatic approach: Use DQX Python classes to run the profiler and quality checks generator on DataFrames or tables, and save results to various storage locations.
    3. No-code approach (Profiler Workflow): If DQX is installed as a workspace tool, you can run the profiler as a workflow that automatically saves generated checks to a file or table.

    Note: Data profiling is typically a one-time action used to discover initial rule candidates. Candidates should be manually reviewed before application. It is not intended to be a continuously scheduled process.

  8. Approaches for applying quality checks in DQX

    main

    DQX provides three primary ways to apply quality checks depending on your technical requirements and data state:

    1. DQX Studio (No-code UI): The recommended option for a no-code experience. It allows you to apply checks to data-at-rest and review results via a browser without writing code or managing configuration files.
    2. Programmatic approach: Used to apply checks programmatically to DataFrames or Tables. This is the required method for performing data quality checks on data-in-transit.
    3. No-code approach (Workflows): Suitable for data already stored in Delta tables or files. This approach applies checks to data-at-rest using workflows if DQX is installed in the workspace as a tool, requiring no code-level integration.

  9. Navigate the DQX Studio home page

    main

    The home page provides a quick overview and two main entry points:

    • Get Started: Initiates the rule creation flow.
    • View Rules: Opens the library of active rules.

    Use the sidebar to navigate functional areas. Note that menu items (such as Run Rules) are automatically hidden if your current role/privileges do not grant access to them.

  10. DQX Studio Technology Stack

    main

    DQX Studio is built using a modern full-stack architecture:

    Backend

    • Runtime: Python 3.12+
    • API Framework: FastAPI (~0.119)
    • Data Validation: Pydantic 2
    • Databricks Integration: Databricks SDK (~0.120) and Databricks SQL Connector (4.2.5) for data-plane queries
    • Database: psycopg 3 (used for Lakebase/Postgres)

    Frontend

    • Framework: React 19 with TypeScript
    • Routing & Data Fetching: TanStack Router and React Query
    • UI Components: shadcn/ui and Tailwind CSS 4
    • Build Tool: Vite 7
    • Type Safety: orval is used to generate TypeScript types and React Query hooks from the OpenAPI specification.

  11. Overview of DQX Summary Metrics

    main

    DQX captures aggregate statistics about your data quality to help track trends and monitor pipeline health. Summary metrics are collected using Spark's built-in Observation functionality and include three types of metrics:

    1. Built-in Metrics: Automatically calculated for every execution.
    2. Per-Check Metrics: A breakdown of error and warning counts for every applied check.
    3. Custom Metrics: User-defined metrics created using Spark SQL expressions.

    These metrics can be persisted to tables for historical analysis.

  12. What is the DQX MCP Server

    main

    The DQX MCP Server is a Model Context Protocol (MCP) server that exposes DQX data quality tools to AI agents. It is deployed as a Databricks App within your workspace and uses on-behalf-of (OBO) authentication. This means every operation performed by an AI agent respects the Unity Catalog permissions of the specific user calling the tool.

    Recommended Use Cases

    • AI Agents: Using Agent Bricks or Genie Code to profile tables, generate rules, and run quality checks programmatically.
    • IDE Workflows: Using Cursor or Claude Code to access DQX tools directly within your development environment.
    • MCP-Compatible Clients: Any client requiring data quality capabilities without direct DQX code integration.