Great Expectations (GX Core)

You can automate responses to data validation outcomes by using Checkpoints. Checkpoints allow you to define a suite of expectations and then trigger specific actions—such as sending alerts or updating Data Docs—based on the resulting ValidationResult.

Key workflows include:

• Creating a Checkpoint with Actions: Configuring a Checkpoint to execute specific tasks when validation succeeds or fails.
• Creating custom Actions: Defining your own logic to run in response to validation results.
• Choosing a Result Format: Configuring how much detail (verbosity) is included in the returned ValidationResult.
• Running a Checkpoint: Executing the validation process to trigger the configured automation.

Great Expectations allows you to add domain-specific knowledge and precise evaluation criteria to your data validations through several customization methods:

• Restrict an Expectation to specific rows: Filter your data so that only a subset of rows are validated for a given Expectation.
• Define a custom Expectation class: Subclass an existing Expectation to add additional notes, metadata, or default attributes.
• Use SQL to define a custom Expectation: Create an Expectation that evaluates the results of a customized SQL query.
• Define a Multi-source Expectation: Create an Expectation that queries multiple Data Sources and compares their results for equality.

GX Core enables you to define and validate data quality checks across several key dimensions. You can use GX to address the following data quality and observability scenarios:

• Distribution: Validate that data values adhere to expected ranges.
• Freshness: Verify that data is available within expected timeframes.
• Integrity: Validate relationships and dependencies between data.
• Missingness: Identify gaps in data to maintain data completeness.
• Schema: Verify that data structure conforms to established rules.
• Uniqueness: Validate that distinct values are present only once.
• Volume: Validate that record quantity falls within expected bounds.
• Unstructured data: Validate structured metadata generated from unstructured content.

Great Expectations (GX) allows you to connect to various data sources and organize that data into Batches for validation. You can connect to data through three primary methods:

• SQL Data: Connect to data residing in SQL databases and organize it into Batches.
• Filesystem Data: Connect to data stored as files within a folder hierarchy and organize it into Batches.
• Dataframes: Connect to data held in memory using pandas or Spark Dataframes and organize it into Batches.

Great Expectations (GX) project settings allow you to manage advanced configurations including metadata storage, Data Docs hosting, credential security, and analytics.

Key configuration areas include:

• Metadata Stores: Define where GX saves Expectation Suite configurations, Checkpoint Configurations, and Validation Results.
• Data Docs: Configure the content and hosting locations for your Data Docs sites.
• Credentials: Securely store and access credentials for database connection strings and tokens for Checkpoint Actions.
• Secrets Managers: Integrate with AWS Secrets Manager, Google Cloud Secret Manager, or Azure Key Vault to retrieve credentials securely.
• Analytics Events: Control whether GX collects internal analytics events.

Data distribution analysis focuses on the spread, shape, and characteristics of data. Validating distribution is critical for ensuring statistical validity for ML models, detecting anomalies/outliers, maintaining data consistency over time, and performing quality control to detect corruption or system errors.

Key aspects to validate include:

• Shape: Using metrics like KL divergence.
• Center: Using metrics like Mean or Median.
• Spread: Using metrics like Standard Deviation or Quantiles.
• Bounds: Using range checks on individual values or summary statistics.

The Great Expectations (GX) documentation provides several learning paths to help you master data quality management:

• GX in your data pipeline: Guidance on where to integrate GX within your existing data workflows to monitor and manage data quality.
• Data quality use cases: Practical scenarios and how to apply GX to address specific data quality challenges.
• Integration tutorials: Step-by-step instructions for integrating GX with other tools in your stack.
• Glossary: A comprehensive alphabetical list of GX-specific terms and definitions to help you understand the core vocabulary.

The great_expectations_contrib tool provides a workflow for managing the lifecycle of contributor packages. Its primary functions are:

• Project Initialization: Creates the standard project structure using a template.
• Completeness Checks: Performs automated checks to ensure your contribution meets the required standards before submission.
• PyPi Upload: Facilitates uploading finished packages to PyPi (requires a pre-existing PyPi account).

A Data Context is the central hub for all Great Expectations (GX) operations. It defines the storage location for your metadata, including:

• Configurations: Data Sources, Expectation Suites, Checkpoints, and Data Docs.
• Validation Results: The outcomes of your data validations and their associated metrics.

Every script that utilizes GX Core should begin by creating a Data Context to provide access to these objects and helper functions.

A ValidationDefinition is a fixed reference that links a specific Batch of data to an Expectation Suite.

It serves two primary purposes:

1. Standalone Validation: It can be run independently to validate a specific batch of data against its associated expectations, which is useful for testing or data exploration.
2. Checkpoint Integration: Multiple ValidationDefinition instances can be passed to a Checkpoint. When the Checkpoint runs, it executes defined Actions based on the validation results for each provided definition.