Great Expectations (GX)

The great_expectations_semantic_types_expectations package provides a collection of Expectations designed to validate data based on its semantic meaning rather than just its technical data type.

While standard expectations might check if a column is a STRING, Semantic Type expectations allow you to validate the actual information the string represents, such as US State abbreviations, ZIP codes, phone numbers, or email addresses. This enables more explicit, fit-for-purpose testing (e.g., asking "Does this column contain valid email addresses?" instead of checking against a manual list of regex patterns or sets).

Great Expectations (GX) allows you to configure advanced project settings to control how metadata is stored, how credentials and secrets are managed, how Data Docs are hosted, and how analytics are handled.

Key configuration areas include:

• Metadata Stores: Specify locations for Expectation Suite configurations, Checkpoint Configurations, and Validation Results.
• Data Docs: Configure the hosting locations and the specific content of your Data Docs sites.
• Credentials: Securely store and access credentials for database connection strings and tokens for Checkpoint Actions.
• Secrets Managers: Integrate with AWS Secrets Manager, Google Cloud Secret Manager, or Azure Key Vault to retrieve credentials securely.
• Analytics Events: Control whether GX collects and sends internal analytics events.

The great_expectations_ethical_ai_expectations package provides a collection of Expectations designed to validate data for degradation, bias, and other Ethical Data concerns. It helps translate ethical codes of conduct into executable code to ensure automated systems (AI/ML models) adhere to fundamental values and avoid discriminatory outcomes.

# Example concept: using an expectation to guard against model degradation
# Note: The specific implementation details depend on the package API
expect_column_label_and_predicted_pair_accuracy_to_be_between

You can add domain and internal knowledge to Expectations to express more specific criteria and precise evaluation for your data. There are four primary ways to customize how Expectations are applied or defined:

1. Restrict an Expectation to specific rows: Filter your data so that only a subset of rows are validated for a given Expectation.
2. Define a custom Expectation class: Create a new Expectation class with additional notes and default attributes by subclassing an existing Expectation.
3. Use SQL to define a custom Expectation: Create an Expectation that operates by evaluating the results of a customized SQL query.
4. Define a Multi-source Expectation: Create an Expectation that queries multiple Data Sources and compares the results for equality.

Filesystem data refers to data stored in file formats like .csv or .parquet within a folder hierarchy (e.g., local filesystems, Amazon S3, Azure Blob Storage, or Google Cloud Storage). Great Expectations (GX) uses either pandas or Spark to read this data.

To work with filesystem data, you follow a three-step hierarchy:

1. Data Source: Defines the location where your data files reside.
2. Data Asset: Configures specific sets of records within a Data Source that you want to access.
3. Batch Definition: Defines how to retrieve records from a Data Asset, either by requesting all records or by partitioning them based on a date/time field.

Great Expectations (GX) allows you to connect to various data sources and organize that data into Batches for validation. Depending on your data storage, you can use one of the following connection patterns:

• SQL Data: Connect to data residing in SQL databases and organize it into Batches.
• Filesystem Data: Connect to data stored as files within a folder hierarchy and organize it into Batches.
• Dataframes: Connect to data held in memory using pandas or Spark Dataframes and organize it into Batches.

Data Sources in Great Expectations (GX) define the location of your data and the connection parameters required to access it. For Filesystem-based data, you direct GX to a specific folder or an online storage location containing your data files.

GX supports the following filesystem-based data source types:

• Local or networked filesystems: Accessing files on your local machine or via network protocols.
• Amazon S3: Accessing data stored in Amazon S3 buckets.
• Azure Blob Storage: Accessing data stored in Azure Blob containers.
• Google Cloud Storage (GCS): Accessing data stored in Google Cloud buckets.

Great Expectations uses Checkpoints to automate responses to Validation Results. Instead of just running validations, you can configure Checkpoints to trigger specific actions—such as sending alerts or updating Data Docs—automatically based on whether the data passed or failed validation.

Key workflows include:

• Creating a Checkpoint with Actions: Configuring built-in actions to run upon completion.
• Creating a custom Action: Defining your own logic to execute based on the validation outcome.
• Choosing a Result Format: Configuring the verbosity of the returned Validation Results.
• Running a Checkpoint: Executing the validation process to trigger the configured automation.

Great Expectations (GX) allows you to define and validate data quality checks across several key dimensions. You can implement checks for the following scenarios to build a comprehensive data quality strategy:

• Distribution: Validate that data values adhere to expected ranges.
• Freshness: Verify that data is available within expected timeframes.
• Integrity: Validate relationships and dependencies between data.
• Missingness: Identify gaps in data to maintain data completeness.
• Schema: Verify that data structure conforms to established rules.
• Uniqueness: Validate that distinct values are present only once.
• Volume: Validate that record quantity falls within expected bounds.
• Unstructured data: Validate structured metadata generated from unstructured content.