DOCSEARCHbyAlgolia
Submit Docs

Definity Documentation

website·Indexed 33 days ago

https://docs.definity.ai

Enterprise lakehouse optimization and observability platform providing full-stack data observability. Definity offers agents for Apache Spark and dbt to capture lineage, metrics, and execution details, helping users identify resource waste, monitor pipeline health, and optimize data platform performance.

Tokens
27.4K
Snippets
81
Records
194
Agent score
59%

What's inside Definity

  1. Overview of definity SaaS deployment advantages

    The definity SaaS deployment is a fully hosted server monitoring option that allows users to monitor pipelines without infrastructure setup or maintenance. Key benefits include:

    • Rapid Setup: Start monitoring in minutes with minimal configuration to capture pipeline job metrics.
    • Real-time Insights: Gain immediate visibility into job performance, latency, and reliability.
    • Managed Infrastructure: definity handles all backend updates, patching, and scaling.
    • Security & Compliance: Data is transmitted over secure channels with access scoped via API tokens. The service is SoC2 Approved and transmits only metadata and statistics, not actual pipeline data.

  2. Overview of Definity observability and optimization platform

    Definity is an intelligent observability and optimization platform designed for Spark-based data pipelines. It provides real-time visibility, predictive insights, and automated recommendations across the data application lifecycle, focusing on pipeline health, performance, data quality, and release validation.

  3. Overview of Definity platform capabilities

    Definity provides a centralized observability platform for production data pipelines that requires zero code changes. It enables data teams to optimize costs via configuration and code recommendations, protect against SLA misses and bad data in real time, and validate infrastructure upgrades using automated CI/CD testing on full-scale data.

  4. Overview of Definity Execution Engine Agents

    Definity uses agents as a connection layer between the platform and various compute environments. These agents allow users to orchestrate, execute, and monitor data workflows across different engines. Currently supported agents include Spark, DBT, and Databricks Serverless, each designed to handle specific types of data processing workloads.

  5. Overview of Definity Spark Agent capabilities

    The Definity Spark Agent provides automatic observability for Apache Spark applications. Key capabilities include:

    • Automatic Lineage Tracking: Captures data lineage across tables, files, and queries.
    • Execution Metrics: Monitors performance, resource usage, and time-series metrics.
    • Custom Metrics: Allows reporting of domain-specific KPIs and data quality measurements.
    • Multi-Task Support: Tracks shared Spark clusters and logical task boundaries.
    • Skew Detection: Identifies and diagnoses data skew issues.
    • Platform Integration: Native support for Databricks, Amazon EMR, and Google Cloud Dataproc.

  6. Overview of definity AI-Powered Data Quality Monitoring

    definity provides real-time data quality monitoring by running agents within pipeline tasks. These agents extract metrics from input, interim, and output tables during execution, allowing for immediate reactions to quality issues. The system uses AI and ML to learn pipeline behavior and automatically generate data quality tests to ensure data accuracy, completeness, and consistency.

  7. Overview of Definity Model Context Protocol (MCP) Integration

    Definity provides a Model Context Protocol (MCP) server that allows AI assistants (such as Claude Desktop or Cline) to securely connect to the Definity data observability platform. This integration enables AI agents to programmatically query pipeline metadata, execution history, data quality metrics, anomalies, lineage information, and task performance.

  8. Understand the Definity data hierarchy

    Definity organizes data work into a four-level hierarchy to track lineage and performance:

    • Pipeline: A complete data application (e.g., an ETL job) consisting of a set of Tasks executed together, typically via an orchestrator like Apache Airflow, Databricks Workflows, or Oozie.
    • Task: An individual computation job within a Pipeline (e.g., a Spark application, DBT project, or Python script). This is the unit of execution instrumented by Definity.
    • Transformation (TF): A single logical operation within a Task, such as an SQL query, a Spark action (e.g., DataFrame.write()), or a DBT model. Definity captures inputs, outputs, duration, and metrics for each TF.
    • Dataset: A data artifact (table, file, or path) read or written by a Transformation. This enables column-level task-dataset lineage.

  9. Monitor data pipeline health with Definity

    Definity provides a Health Monitoring system for data pipelines to ensure reliability and performance. It continuously tracks three primary health areas:

    1. Pipeline Executions & Success Rate: Monitors successful vs. failed executions to identify trends.
    2. Task Failures & Retries: Tracks internal pipeline failures to assess stability.
    3. SLA Compliance & Execution Duration: Measures adherence to SLAs and identifies performance bottlenecks at the pipeline and task levels.

  10. Trace data dependencies with Column-Level Lineage

    Definity provides column-level lineage to map transformations down to individual column dependencies. This capability allows developers and data engineers to trace specific column origins across tasks, identify how schema changes impact downstream consumers, and understand precise dependencies between datasets and pipelines.

  11. Track historical data states using Point-In-Time (PIT) Lineage

    Definity supports Point-In-Time (PIT) lineage, which connects dependencies at the logical execution level based on the input data snapshot. This allows users to trace lineage across different pipeline execution times and historical data states, rather than just seeing the current state of the pipeline.

  12. Monitor pipeline Execution Health with Definity

    Definity monitors job health by tracking failures, retries, SLA misses, disk spills, long idle times, and skew patterns. When a health pattern becomes consistent across multiple runs, Definity generates an insight to notify the owner and suggest a resolution.