DOCSEARCHbyAlgolia
Submit Docs

pandera

repository·main·Indexed 26 days ago

https://github.com/unionai-oss/pandera

A light-weight and flexible data validation and testing tool for statistical data objects. Pandera provides an expressive API for performing data validation on dataframe-like objects, including pandas, polars, pyspark, and dask. It supports both object-based (DataFrameSchema) and class-based (DataFrameModel) APIs, offering built-in and custom checks, vectorized validation, and an optional Narwhals-powered backend to unify validation paths.

Tokens
90.5K
Snippets
248
Records
430
Agent score
88%

What's inside pandera

  1. Overview of Xarray Integration

    main
    Pandera extends its validation capabilities to the N-dimensional array ecosystem by supporting xarray data structures. This allows developers to use a consistent validation API across both tabular (pandas, polars, etc.) and multi-dimensional array data. The integration supports xr.DataArray, xr.Dataset, and xr.DataTree containers, as well as various duck array backends like NumPy, Dask, CuPy, and Sparse.

  2. Validate Xarray objects with Pandera

    main

    Pandera provides schemas and components for validating xarray.DataArray, xarray.Dataset, and xarray.DataTree objects.

    Typical imports use pandera.xarray. The pandera.xarray entry point also re-exports core Pandera components including Check, Parser, decorators (like check_input), and pandera.errors.

  3. Performance optimization for composite checks in data generation

    main

    Pandera's data generation engine (Stage 5 / Stage 8 refactor) optimizes composite checks by merging multiple constraints into single, efficient operations rather than chaining multiple rejection filters. This is particularly effective for:

    • Numeric ranges: Combining Check.gt(min) and Check.lt(max) lowers to a single from_dtype(min_value=min, max_value=max) call.
    • String patterns: Combining Check.str_startswith and Check.str_endswith merges patterns into a single anchored regex (e.g., \A(?:foo).*(?:bar)\Z) driven directly by hypothesis.
    • Xarray support: data_array_strategy and dataset_strategy now correctly respect and aggregate checks arguments.

  4. Explore Pandera Ecosystem Integrations

    main

    Pandera provides integrations with several popular Python libraries to enhance data validation, type safety, and data synthesis. Supported integrations include:

    • FastAPI: Use pandera DataFrameModels within your FastAPI applications.
    • Frictionless: Convert Frictionless schemas into Pandera schemas.
    • Hypothesis: Use the Hypothesis library to generate valid data that adheres to your Pandera schema constraints.
    • Mypy: Perform static type-linting on your pandas and pandera code (Experimental 🧪).
    • Pydantic: Use pandera DataFrameModels when defining your Pydantic BaseModels.

  5. Understand Pandera Data Synthesis Architecture

    main

    Pandera's data synthesis layer uses hypothesis to generate data (DataFrames, Series, Indexes, or xarray containers) that satisfy a schema's type and Check constraints. This is used by the Schema.strategy() and Schema.example() APIs, as well as pa.Field(...) generation.

    Current Implementation Details

    • Chained Strategy Pattern: Currently, each Check contributes a hypothesis strategy by applying .filter() calls on top of the previous strategy.
    • Performance Implications: Because of filter chaining, multiple constraints (e.g., Check.gt(0) and Check.lt(100)) result in a strategy that draws values and then discards them via filters. This can lead to slow performance and frequent Unsatisfiable errors or Hypothesis health-check triggers (filter_too_much, data_too_large).
    • Constraint Order: In the current implementation, users are often advised to place the most restrictive constraint first to minimize the impact of filter chaining.

  6. Understand Pandera Data Types

    main

    Pandera uses its own DataType interface to abstract data structures like Apache Spark, Apache Arrow, and xarray. This allows for a standardized API and the ability to define logical data types (e.g., IPAddress) on top of physical data types (e.g., str).

    Key components:

    • pandera.dtypes: Defines semantic types for framework engines.
    • pandera.engines.numpy_engine: Implements NumPy datatypes.
    • pandera.engines.pandas_engine: Implements Pandas-specific datatypes (e.g., pd.DatetimeTZDtype).

    Users can extend the interface by modifying data type checks, modifying the behavior of the coerce argument in DataFrameSchema, or adding custom data types.

  7. Summary of Optimized Data Synthesis Strategies

    main

    The optimized data synthesis architecture introduces a refactor designed to improve performance and error handling during schema construction. Key improvements include:

    • Reduced Overhead: Uses a single npst.from_dtype / st.sampled_from / st.from_regex call per field instead of multiple filters.
    • Early Error Detection: Joint-unsatisfiability is now raised as a SchemaDefinitionError during strategy construction, rather than an Unsatisfiable error after hypothesis has exhausted its search.
    • Improved xarray Support: xarray now correctly honors Checks during synthesis.
    • Optimized Check Ordering: Built-in check ordering is no longer a performance concern.

    Compatibility Note: All existing user code, custom strategies, and extension entry points remain unchanged and fully compatible with this refactor.

  8. Supported DataFrame Libraries

    main

    Pandera supports direct validation for several core DataFrame libraries:

    • Pandas: The original supported library for validating pandas dataframes.
    • Polars: High-performance validation for Polars dataframes.
    • Ibis: Validation for Ibis tables (portable Python dataframe library).
    • PySpark SQL: Validation for large-scale data processing using PySpark SQL.

  10. Pandera API Overview for DataFrames

    main

    Pandera provides several specialized API modules for validating tabular and array-like data:

    • Core: The primary objects used for defining schemas.
    • GeoPandas: Entry points for spatial data using GeoDataFrameSchema and GeoDataFrameModel.
    • Data Types: Tools for type checking and data coercion.
    • DataFrame Models: A class-based API for defining types for tabular or array-like data.
    • Decorators: Tools to integrate pandera schemas directly into Python functions.
    • Schema Inference: Capabilities to bootstrap schemas automatically from existing real-world data.
    • IO Utilities: Functions for reading and writing schema definitions.
    • Data Synthesis Strategies: Functions for generating synthetic data based on defined schemas.
    • Narwhals Backend: An opt-in backend that unifies validation paths for Polars, Ibis, and PySpark SQL.
    • Extensions: Utility functions to extend pandera's core functionality.
    • Errors: A collection of pandera-specific exceptions for debugging validation failures.

  12. Participate in data synthesis with in-line `pa.Check` predicates

    main

    By default, opaque user predicates (e.g., pa.Check(lambda x: ... )) are treated as filter-based and fall through to residual_filters. To make an in-line check participate in the optimized aggregator for data synthesis, you must explicitly provide a constraint adapter.

    You can do this in two ways:

    1. Attach a constraint= adapter directly to the Check object.
    2. Provide an adapter via register_check_method(constraint=) or register_check_constraint.