DOCSEARCHbyAlgolia
Submit Docs

PDQ Documentation

website·Indexed 29 days ago

https://simplitics-docs.vercel.app

Documentation for the PDQ platform, a metadata-driven data orchestration system. It covers the Active Metadata Engine (AME) API, the INGEST extraction service, Data Lake Service (DLS), and Data Warehouse Automation (DWA). The platform supports generating SQL for target platforms including Snowflake, Databricks, and PostgreSQL, and includes a DataOps Console for monitoring system health and Quality Performance Indicators (QPI).

Tokens
43.4K
Snippets
69
Records
343
Agent score
75%

What's inside PDQ

  1. Overview of the PDQ Layered Data Platform Architecture

    PDQ utilizes a four-stage layered architecture to move data from raw sources to quality-monitored warehouse models:

    1. INGEST (Ingestion): Extracts data from source systems (databases, APIs, files) and lands it in the data lake.
    2. DLS (Data Lake Service): Organizes, archives, validates, and publishes raw data through a series of storage zones.
    3. DWA (Data Warehouse Automation): Models, maps, and transforms data into structured warehouse models using a Stage → Base → Core strategy.
    4. QPI (Quality Performance Indicator): Monitors data health, mapping coverage, and pipeline quality across all layers.

  2. Overview of PDQ by Simplitics Platform Components

    PDQ by Simplitics is a unified architecture for data management automation. The platform consists of several integrated components that handle metadata, ingestion, storage, warehousing, and quality control:

  3. Overview of the DataOps Console

    The DataOps Console is the central control center for monitoring runtime operators and statistics. It provides runtime slicing and dicing for process inspection and allows for the safe restart of idempotent tasks. The console is organized by platform stage:

    • Ingestion
    • Data Lake
    • Data Warehouse
    • Data Quality
    • Reference
    • System

  4. Overview of the DataOps Console

    The DataOps Console is an operator-facing web application for the PDQ platform used for monitoring and intervention. It allows operators to track scheduled ingestions, data lake movements, warehouse loads, and quality checks, and provides the ability to restart, skip, or re-trigger failed work. It is not a configuration interface; pipelines are defined by metadata in the PDQ repository, which the console reads via the PDQ REST API.

  5. Monitor DWA task loading and progress

    The DWA console provides several views to monitor data loading:

    • State Metrics: A counter row showing Expected, Completed, Queued, Running, Scheduled, and Failed tasks.
    • Tasks still in progress: A colored bar chart and table of incomplete work. Users can filter by state and select specific rows to Restart or Skip.
    • DWA Task loading overview: Provides a Gantt chart (Overview tab) showing execution duration and a table (Details tab) with millisecond timestamps.
    • DWA Loading Task checklist: Compares scheduled sourcefiles against actual starts, highlighting rows that never started.

  6. PDQ Architecture Overview

    PDQ is a containerized system deployed in a strict linear sequence to ensure stability. The core components include:

    • AME (Active Metadata Engine): The central command center and blueprint archive that defines system purposes via APIs.
    • Workers (INGEST, DLS, DWA): Specialized containerized units for data importing, lake services, and warehouse automation.
    • Caddyserver: The secure gatekeeper providing authentication (via Entra ID) and protecting internal systems from external interference.

  7. Implement secure authentication for Data Lake Service v3.2

    v3

    Data Lake Service v3.2 introduces several security and credential management features:

    • Snowflake key-pair (JWT) authentication.
    • Dynamic credential resolution via AWS Secrets Manager and Azure Key Vault.
    • An offline key-verification tool for validating credentials.

  8. Supported target platforms for PDQ SQL generation

    PDQ generates SQL from a single consistent model for the following six target platforms: Snowflake, Databricks, SQL Server, Azure Synapse, Microsoft Fabric, and PostgreSQL. While the model remains the same, the generated SQL dialect varies based on the capabilities of the target engine.

  9. Configure INGEST task view filters and limits

    The INGEST task overview provides several controls to manage the displayed data:

    • Max records slider: Caps the history fetched for 'Completed' rows (default is 20,000). Note: Failed and still-running tasks are always shown in full and are not affected by this cap.
    • Search: Matches against Export or System names.
    • Multiselect Filters: Filter by Type, State, and Scheduled At.
    • View Tabs:
      • Overview: A Gantt chart of task duration and timing with 1d / 7d / All zoom levels.
      • Details: A tabular view of the same task data.

  10. Understand the Simplitics Logical Data Architecture

    Simplitics organizes data flow through five distinct architectural layers to minimize code drift and optimize total cost of ownership (TCO):

    1. Landing Zone: The initial ingestion layer for raw data.
    2. Raw Archived Data: A storage layer dedicated to historical data.
    3. Published Data: Processed data ready for usage, utilizing source context and source-oriented naming conventions.
    4. Integrated Model: A unified model that maps data from various sources into common core business concepts.
    5. Business-Oriented Data Product: The final layer where data is organized into specific products for end-user consumption.

  11. Understand the v3.2 Versioning and AME API Compatibility

    v3
    Starting with version 3.2, version numbers track the Active Metadata Engine (AME) API contract required by a component. A component and a backend are compatible if their version numbers agree. For example, a 3.2 component requires the v3 API family and uses object-level generation on /api/v3.2. Upgrading the backend to v3 is the required first step of any upgrade to ensure compatibility and prevent runtime failures.

  12. Understand the PDQ data architecture flow

    PDQ uses a linear, stage-based architecture where each layer has a specific responsibility to ensure faults are locatable and layers are replaceable. The flow is as follows:

    1. Sources: Raw data from files, APIs, streams, and databases. Defined via metadata (connection, schema, load window).
    2. Ingest: The Ingest Agent pulls data and lands it unchanged to maintain a reconciliation trail.
    3. Data Lake Service (DLS): Moves data through Landing → Raw Archive → Trusted. Profiling and quality rules run here; only critical failures (e.g., missing business keys) stop the flow.
    4. Data Warehouse Automation (DWA): Moves data through Published → Integrated → Business. The 'Integrated' layer is a normalized, historized model where load logic and point-in-time views are auto-generated from the model.
    5. Delivery: Exposes business-facing views for BI reports, APIs, AI/ML workflows, and SQL access.