Great Expectations Documentation

website·Indexed 39 days ago

https://docs.greatexpectations.io/docs/

Documentation for Great Expectations (GX), a tool for validating and documenting data. Includes guides on creating Data Contexts, configuring credentials, and connecting to SQL, Spark, and cloud storage data sources. Provides API references for ExpectationSuites, ValidationDefinitions, and various datasource classes for versions 0.18 and 1.23.1.

Tokens: 246.7K
Snippets: 1.3K
Records: 1.9K
Agent score: 68%

What's inside Great Expectations

Overview of Great Expectations Stores
A Store is a connector used to store and retrieve metadata within Great Expectations. Stores are accessed via the Data Context. There are several specialized store types tailored to specific metadata:
https://docs.greatexpectations.io/docs/0.18/reference/learn/terms/store/
Overview of AbstractDataContext CRUD capabilities
The great_expectations.data_context.AbstractDataContext class is the base class for all Data Contexts. It encapsulates store and core components and is primarily responsible for managing CRUD (Create, Read, Update, Delete) operations for core Great Expectations objects.
Capabilities by object type:
- Datasources, ExpectationSuites, Checkpoints, Profilers: Supports get, add, update, add_or_update, and delete.
- Stores: Supports add and delete only.
https://docs.greatexpectations.io/docs/0.18/reference/api/data_context/abstractdatacontext_class/
Overview of RegexBasedColumnMapExpectations
RegexBasedColumnMapExpectations are a sub-type of ColumnMapExpectation used for regex-powered validation of a single column. They evaluate every row in a column against a regex pattern and calculate the percentage of positive matches. The Expectation is considered valid if this percentage meets a specified threshold (100% by default). This threshold is controlled via the mostly parameter, which accepts a float between 0 and 1.
https://docs.greatexpectations.io/docs/0.18/oss/guides/expectations/creating_custom_expectations/how_to_create_custom_regex_based_column_map_expectations/
Overview of Great Expectations Checkpoints
A Checkpoint is the primary mechanism for validating data in a production deployment. It acts as an abstraction that bundles the validation of one or more Batches of data against one or more Expectation Suites, and defines the Actions to be taken after validation (e.g., sending Slack notifications or updating Data Docs). Checkpoints are managed by a Data Context and their configurations are persisted as YAML files, making them reusable and version-controllable.
https://docs.greatexpectations.io/docs/0.18/reference/learn/terms/checkpoint/
Overview of MulticolumnMapExpectations
MulticolumnMapExpectations are a specialized type of Expectation in Great Expectations (v0.18) used to verify row-wise relationships between a set of columns. They evaluate a yes/no question for each row across the specified columns and calculate the percentage of rows that meet the criteria. The data is considered valid if this percentage meets a defined threshold.
https://docs.greatexpectations.io/docs/0.18/oss/guides/expectations/creating_custom_expectations/how_to_create_custom_multicolumn_map_expectations/
Overview of Great Expectations Agent Skills
GX bundles 'agent skills'—packaged guidance files that AI coding agents (Claude Code, Codex, and Cursor) read to operate the GX public Python API. These skills allow users to describe data quality goals in plain language, which the agent then implements using the Python API. The result is a standard GX project and a script that can be run independently of the agent.
https://docs.greatexpectations.io/docs/core/set_up_a_gx_environment/install_agent_skills/
Overview of Expectation Suites in GX Cloud
Expectation Suites are collections of multiple Expectations associated with a single Data Asset. They are used to understand data and improve data quality. By default, a default Expectation Suite is automatically created whenever a new Data Asset is created.
https://docs.greatexpectations.io/docs/0.18/cloud/expectation_suites/manage_expectation_suites/
Overview of FileDataAsset class
The FileDataAsset class is the base class for PathDataAssets in Great Expectations. It is used to define data assets that create batches by applying regular expressions to file names.
```
from great_expectations.datasource.fluent.data_asset.path.file_asset import FileDataAsset

# Example instantiation
asset = FileDataAsset(
    name="my_file_asset",
    type="CSV",
    connect_options={
        "path": "s3://my-bucket/data/"
    }
)
```
https://docs.greatexpectations.io/docs/reference/api/datasource/fluent/data_asset/path/file_asset/filedataasset_class/
Overview of ColumnPairMapExpectations
ColumnPairMapExpectations are a specialized sub-type of Expectation in Great Expectations (v0.18). They evaluate a row-wise relationship between a pair of columns to answer a yes/no question. The Expectation calculates the percentage of rows that satisfy the condition; if this percentage meets a specified threshold, the data is considered valid.
https://docs.greatexpectations.io/docs/0.18/oss/guides/expectations/creating_custom_expectations/how_to_create_custom_column_pair_map_expectations/
Overview of Great Expectations for data validation
Great Expectations is a tool for validating and documenting data, acting as 'unit tests for data'. It allows data engineering and science teams to assert expectations about data during ingestion or transformation to prevent quality issues from reaching data products. It converts these assertions into human-readable data documentation and quality reports.
https://docs.greatexpectations.io/docs/0.18/core/introduction/introduction/
Overview of the GX Core data validation workflow
GX Core is a Python library used to build programmatic data validation workflows. All GX workflows follow a consistent four-step pattern:
1. Set up a GX environment: Initialize the Data Context.
2. Connect to data: Define Data Sources, Data Assets, and Batch Definitions.
3. Define Expectations: Create Expectations and organize them into Expectation Suites.
4. Run Validations: Associate data with expectations via Validation Definitions and execute them using Checkpoints.
https://docs.greatexpectations.io/docs/core/introduction/gx_overview/
Overview of GX Cloud for data quality
GX Cloud is a data quality solution designed to provide comprehensive test coverage and proactive data quality monitoring. It allows teams to catch data issues before they impact downstream processes. Key capabilities include collaborative input from stakeholders, shared accessibility of results, and the ability to integrate into existing data pipelines via a rich API (supporting Python and Jupyter notebooks) and various orchestrators.
https://docs.greatexpectations.io/docs/0.18/cloud/why_gx_cloud/