DOCSEARCHbyAlgolia
Submit Docs

dataframely

repository·main·Indexed 20 days ago

https://github.com/quantco/dataframely

A declarative Python library for validating the schema and content of Polars DataFrames. It provides tools to define schemas using BaseSchema and Column types, enforce relationship constraints, and apply row-wise or aggregate validation rules. The library includes support for type casting, primary key uniqueness, and integration with Pydantic for validating Polars DataFrames and LazyFrames within models.

Tokens
32K
Snippets
122
Records
156
Agent score
69%

What's inside dataframely

  1. Overview of Dataframely

    main

    Dataframely is a Python package designed to validate the schema and content of polars data frames. It aims to make data pipelines more robust by ensuring data meets specific expectations and more readable by providing schema information through data frame type hints.

    Key capabilities include:

    • Declarative schema definitions using classes.
    • Column-specific validation (e.g., nullability, string length).
    • Cross-column and group validation, including primary key checks.
    • Validation across collections of interdependent data frames.
    • Soft validation (filtering out invalid rows instead of raising errors).
    • Introspection of validation failures.
    • Enhanced type hints for data pipeline contracts.
    • Integration with sqlalchemy, pydantic, and Arrow PyCapsule.
    • Generation of compliant test data.

  2. Use inline_for_sampling to simplify override syntax

    main

    By default, overriding values for a collection member requires nesting them under the member's name (e.g., {"invoice": {"amount": 1000.0}}).

    You can simplify this by declaring a collection member as "inlined for sampling" using the dy.CollectionMember type annotation with inline_for_sampling=True. This allows you to supply non-primary key columns directly at the top level of the override dictionary.

    Use Annotated from typing to apply this configuration to your schema members.

    from typing import Annotated

class HospitalInvoiceData(dy.Collection):
    invoice: Annotated[
        dy.LazyFrame[InvoiceSchema],
        dy.CollectionMember(inline_for_sampling=True),
    ]
    diagnosis: dy.LazyFrame[DiagnosisSchema]

# Now you can override 'amount' directly at the top level
HospitalInvoiceData.sample(overrides=[
    {
        "invoice_id": "1",
        "amount": 1000.0,  # This is now valid because of inline_for_sampling
        "diagnosis": [{"code": "E11.2"}],
    }
])

  3. Understand FilterResult and its variants

    main

    In dataframely, filtering operations return a result object that encapsulates the outcome of the filter. Depending on the context and the underlying data structure, you will encounter one of the following types:

    • FilterResult: The standard result object returned after a filtering operation.
    • LazyFilterResult: A result object that performs filtering lazily, potentially deferring computation until the data is explicitly accessed.
    • CollectionFilterResult: A specialized result object used when filtering collections.

  4. Understand error behavior for eager vs lazy validation

    main

    The type of error raised during validation depends on the eager parameter setting:

    • eager=True (Default): The validate function (for both Schema and Collection) raises a dataframely.ValidationError immediately.
    • eager=False: No error is raised during the call. Instead, a polars.exceptions.ComputeError is raised later when the collect() method is called on the lazy frame.

    Important Caveats

    • Collection Error Messages: When eager=False, error messages for Collection.validate are limited and non-deterministic; they may only report information about a single (randomly selected) failing member.
    • Streaming Engine: If collecting a lazy frame using the Polars streaming engine, validation may abort at the first failure encountered, making the specific failure reported non-deterministic across executions.

  5. String column length handling in SQL generation

    main

    When generating SQL for string columns, dataframely attempts to include maximal length constraints:

    • If max_length is explicitly set, it is used in the SQL definition.
    • If a regex is provided, the maximal length is inferred from the regular expression if possible.

    This is particularly important for primary key columns in databases like Microsoft SQL Server, which do not allow unbounded VARCHAR(max) columns to serve as primary keys.

  6. Apply column-level and cross-column constraints in `dy.Schema`

    main

    Constraints allow you to persist implicit data assumptions directly in your schema.

    1. Column-level Constraints

    Use pre-defined arguments like nullable, min, or regex. For non-standard constraints, use the check argument with a dictionary mapping a name to a lambda/function.

    class MySchema(dy.Schema):
    # Using check for custom logic
    col = dy.UInt8(check={"divisible_by_two": lambda col: (col % 2) == 0})

    2. Cross-column Constraints (Rules)

    Use the @dy.rule() decorator to define constraints involving multiple columns. Use cls to access the columns.

    class MySchema(dy.Schema):
    col1 = dy.UInt8()
    col2 = dy.UInt8()

    @dy.rule()
    def col1_greater_col2(cls) -> pl.Expr:
        return cls.col1.col > cls.col2.col

    3. Cross-row Constraints

    Use rules with an over expression for constraints that span multiple rows (beyond primary key checks).

    class MySchema(dy.Schema):
    col = dy.UInt8(check={"divisible_by_two": lambda col: (col % 2) == 0})

class MySchema(dy.Schema):
    col1 = dy.UInt8()
    col2 = dy.UInt8()

    @dy.rule()
    def col1_greater_col2(cls) -> pl.Expr:
        return cls.col1.col > cls.col2.col

  7. How fuzzy sampling works with rules and constraints

    main

    When a schema contains custom @dy.rules or primary_key constraints, Dataframely uses "fuzzy sampling". It samples data in a loop until it finds a dataset of length num_rows that satisfies all constraints.

    Important Considerations:

    • The maximum number of sampling rounds is controlled by dataframely.Config.set_max_sampling_iterations.
    • If the limit is reached without finding valid data, sampling fails.
    • If you set the max iterations to 1, you can only reliably sample from schemas that have no custom rules or primary key constraints.

  8. How primary keys work in a Collection

    main

    A dataframely.Collection unifies multiple tables relating to the same set of underlying entities. This allows dataframely.filter operations to use information from multiple tables to validate an entity.

    Requirement: If you define any dataframely.filters within a Collection, all tables in that collection must have an overlapping primary key. This means there must be at least one column that is marked as a primary key in every table included in the collection.

  9. Understand Dataframely versioning and breaking changes

    main

    Dataframely follows semantic versioning. Breaking changes to user-facing functionality are only introduced in major releases.

    To manage updates safely:

    • Control dependency versions: Use package managers with lockfile support (e.g., pixi) to ensure reproducible environments.
    • Automated testing: Run your test suite when updating lockfiles to verify compatibility with newer versions.
    • Monitor release notes: Check the GitHub releases page for details on new versions.
    • Handle DeprecationWarnings: Dataframely uses Python's DeprecationWarning to signal upcoming breaking changes. It is recommended to migrate code proactively rather than silencing these warnings.

  10. Use experimental features in the dataframely.experimental namespace

    main

    Experimental features are located in the dataframely.experimental namespace.

    Warning: The standard semantic versioning policy does not apply to this namespace. Breaking changes may be introduced to experimental features in minor releases. Use these features at your own risk and expect frequent API changes.

  11. Add custom cross-column rules using `@dy.rule()`

    main

    For validation logic that involves multiple columns (e.g., checking a ratio between two fields), use the @dy.rule() decorator on a method within your Schema class. The method should return a Polars expression (pl.Expr) that evaluates to a boolean for each row.

    If using ruff for linting, add the following to your pyproject.toml to prevent linting errors on the decorator:

    [tool.ruff.lint.pep8-naming]
classmethod-decorators = ["dataframely.rule"]
    import dataframely as dy
import polars as pl


class HouseSchema(dy.Schema):
    zip_code = dy.String(nullable=False, min_length=3)
    num_bedrooms = dy.UInt8(nullable=False)
    num_bathrooms = dy.UInt8(nullable=False)
    price = dy.Float64(nullable=False)

    @dy.rule()
    def reasonable_bathroom_to_bedroom_ratio(cls) -> pl.Expr:
        ratio = pl.col("num_bathrooms") / pl.col("num_bedrooms")
        return (ratio >= 1 / 3) & (ratio <= 3)

  12. Configure nullability for columns and primary keys

    main

    Dataframely v2 has changed the default behavior for nullability:

    1. Columns: By default, columns are now non-nullable (nullable=False). In v1, they were nullable by default with a warning. Nullability is now opt-in.
    2. Primary Keys: Primary key columns may not be nullable. While v1 only issued a warning, v2 will raise an exception if a primary key is designated as nullable.