repository·master·Indexed 21 days ago
https://github.com/awslabs/python-deequPyDeequ is a Python API for Deequ, a library built on Apache Spark for defining 'unit tests for data' to measure data quality in large datasets. It provides tools for computing data metrics via Analyzers, performing data profiling with ColumnProfilerRunner, suggesting constraints via ConstraintSuggestionRunner, and validating data using VerificationSuite and Check objects. The library also supports anomaly detection strategies, row-level result extraction, and persisting metrics using FileSystemMetricsRepository or InMemoryMetricsRepository.
pydeequ.analyzers module provides tools to compute various statistics and metrics about your datasets. Analyzers are used to scan data and produce summary metrics that can be used for monitoring or further analysis.PyDeequ provides specialized classes to handle different data types within a profile:
StandardColumnProfile: General purpose column profile.StringColumnProfile: Specialized profile for string-type columns.NumericColumnProfile: Specialized profile for numeric-type columns.To perform data analysis in PyDeequ, you use the AnalysisRunBuilder to configure your analysis and the AnalysisRunner to execute it.
AnalysisRunBuilder.addAnalyzer(analyzer) to add specific statistical analyzers to your run.AnalysisRunBuilder.useRepository(repository) to specify where results should be stored.AnalysisRunBuilder.saveOrAppendResult(resultKey) to define the key for saving or appending results.AnalysisRunBuilder.run() to execute the analysis.AnalysisRunner.onData(DataFrame) to provide the Spark DataFrame to be analyzed.pydeequ.profiles module enables you to create comprehensive profiles of your datasets. Profiles provide a high-level overview of the data's structure, distribution, and quality metrics.pydeequ.repository module provides abstractions for managing and accessing data repositories, which is useful when working with historical data or structured data storage for quality tracking.pydeequ.anomaly_detection module allows you to identify unexpected patterns or outliers in your data distributions over time, helping to detect data drift or quality issues.pydeequ.suggestions module can automatically suggest data quality constraints based on the observed characteristics of your dataset, helping to automate the creation of data quality tests.pydeequ.checks module is used to define unit tests for your data. You can specify constraints (e.g., column completeness, value ranges, or uniqueness) and verify that your dataset adheres to these rules.pydeequ.verification module provides the mechanisms to execute the checks defined in the checks module and determine if the dataset passes or fails the specified quality requirements.To prevent hanging processes, ensure you shut down the Spark session and the Spark context gateway callback server after your jobs are complete.
spark.sparkContext._gateway.shutdown_callback_server()
spark.stop()When creating or modifying a Glue Development Endpoint to support PyDeequ, you must specify the S3 locations for the libraries in the following fields:
pydeequ.zip file.Ensure that the IAM role associated with AWS Glue has the necessary permissions to access these S3 paths.
To use PyDeequ, you must initialize a SparkSession configured with the necessary Deequ Maven coordinates to ensure the underlying Scala/Java libraries are available. Use pydeequ.deequ_maven_coord and pydeequ.f2j_maven_coord for configuration.
from pyspark.sql import SparkSession, Row
import pydeequ
spark = (SparkSession
.builder
.config("spark.jars.packages", pydeequ.deequ_maven_coord)
.config("spark.jars.excludes", pydeequ.f2j_maven_coord)
.getOrCreate())
# Example DataFrame
df = spark.sparkContext.parallelize([
Row(a="foo", b=1, c=5),
Row(a="bar", b=2, c=6),
Row(a="baz", b=3, c=None)]).toDF()