chispa

When using assert_df_equality with the allow_nan_equality=True flag, the comparison logic extends to array-valued fields. Two arrays are considered equal if they have the same length and contain NaN values at the exact same indexes.

Note that allow_nan_equality=True does not ignore position; NaN values must align at identical indexes for the arrays to be treated as equal. If NaN values appear at different indexes, or if there is a mismatch in non-NaN element values, the assertion will fail.

By default, chispa follows strict equality where NaN != NaN. If your test requires NaN values to be treated as equal (similar to pandas behavior), you must explicitly enable this via the allow_nan_equality flag.

# Default behavior: fails if NaNs are present and not identical
assert_df_equality(df1, df2)

# Opt-in behavior: treats NaN values as equal
assert_df_equality(df1, df2, allow_nan_equality=True)

Chispa performs schema comparisons before analyzing DataFrame content. If the schemas of the two DataFrames being compared do not match (e.g., different column names or different data types), the assertion will fail immediately with a schema mismatch error. This prevents unnecessary and potentially misleading content comparisons when the underlying structures are incompatible.

def test_schema_mismatch_message():
    data1 = [(1, "a"), (2, "b"), (3, "c"), (None, None)]
    df1 = spark.createDataFrame(data1, ["num", "letter"])

    data2 = [(1, 6), (2, 7), (3, 8), (None, None)]
    df2 = spark.createDataFrame(data2, ["num", "num2"])

    # This will fail with a schema mismatch error because column names/types differ
    assert_df_equality(df1, df2)

PySpark schemas distinguish between nullable and non-nullable columns. If you want to compare the data without failing on differences in the nullable property of the schema, use ignore_nullable=True.

assert_df_equality(df_expected, df_actual, ignore_nullable=True)

The core functionality of chispa is assert_df_equality, which compares two PySpark DataFrames. It provides detailed error messages showing exactly where the DataFrames differ (rows, columns, or values).

from chispa.dataframe_comparisons import assert_df_equality

assert_df_equality(df_expected, df_actual)

By default, chispa follows standard Spark behavior where NaN != NaN. If you want to treat NaN values as equal during comparison, opt-in by setting allow_nan_equality=True.

assert_df_equality(df_expected, df_actual, allow_nan_equality=True)

Use chispa to verify that specific columns in two DataFrames contain identical values. This is useful for validating transformations on specific features without comparing the entire schema.

from chispa.dataframe_comparisons import assert_column_equality

assert_column_equality(df_expected, df_actual, column_names=['col1', 'col2'])

If you are contributing to or developing chispa locally, the project uses Poetry for dependency management.

To set up your environment and run tests:

1. Install dependencies: poetry install
2. Run the test suite: poetry run pytest tests

To exclude certain columns from the comparison (such as timestamps, IDs, or metadata columns that change frequently), pass a list of column names to the ignore_columns parameter.

assert_df_equality(df_expected, df_actual, ignore_columns=['timestamp', 'uuid'])

If the columns in your DataFrames are not in the same sequence but contain the same data, use the ignore_column_order=True option.

assert_df_equality(df_expected, df_actual, ignore_column_order=True)

By default, assert_df_equality expects rows to be in the same order. If your DataFrames have the same data but different row ordering (e.g., due to shuffling), use the ignore_row_order=True option.

assert_df_equality(df_expected, df_actual, ignore_row_order=True)

You can install chispa using pip or add it as a development dependency using Poetry.

# Using pip
pip install chispa

# Using Poetry
poetry add chispa --group dev