DOCSEARCHbyAlgolia
Submit Docs

numpydantic

repository·main·Indexed 41 days ago

https://github.com/p2p-ld/numpydantic

Type and shape validation and serialization for arbitrary array types in Pydantic models. It supports backends including NumPy, Dask, Zarr, HDF5, and video files, providing static type checking for array shapes and dtypes via a mypy plugin. Features include NDArray and Shape annotations, lazy loading of large datasets, and the generation of JSON Schemas with shape constraints.

Tokens
19.9K
Snippets
59
Records
84
Agent score
99%

What's inside numpydantic

  1. Serialize NDArray to JSON (Normal Style)

    main

    By default, numpydantic serializes arrays to JSON using the 'Normal Style', which represents n-dimensional arrays as standard JSON lists-of-lists.

    Warning: This method is not suitable for large arrays as it dumps the entire content into the JSON string. Additionally, all dtype information is lost, and numbers are parsed as standard Python int or float types. When re-validated, these arrays will default to being handled by the NumpyInterface.

    from pydantic import BaseModel
from numpydantic import NDArray
import numpy as np

class MyModel(BaseModel):
    array: NDArray

model = MyModel(array=np.array([[1, 2], [3, 4]]))
json_string = model.model_dump_json()

  2. The Interface validation lifecycle

    main

    When validating an array, an interface executes a specific chain of lifecycle methods. If you are implementing a custom interface, you can override these methods to handle specific array formats, dtypes, or shapes.

    Validation Sequence

    1. deserialize: Deserializes data dumped via pydantic.BaseModel.model_dump_json(round_trip=True).
    2. before_validation: A hook to modify input data (e.g., coercion or wrapping in a proxy) before validation begins. It must accept types specified in Interface.input_types.
    3. Dtype Validation Cluster:
      • get_dtype: Retrieves the dtype information.
      • validate_dtype: Validates the dtype.
      • raise_for_dtype: Handles errors/exceptions related to dtype mismatches.
    4. after_validate_dtype: A hook to modify the array or bail early after dtype validation is complete.
    5. Shape Validation Cluster:
      • get_shape: Retrieves the array shape.
      • validate_shape: Validates the shape.
      • raise_for_shape: Handles errors/exceptions related to shape mismatches.
    6. after_validation: A final hook to modify the array before it is assigned to the model field. This method must return an object matching the Interface.return_type.

  3. How interfaces bridge NDArray specifications and array libraries

    main

    Interfaces are the mechanism used to connect the abstract numpydantic.NDArray specification to concrete array libraries (like NumPy, Dask, or Zarr). They are subclasses of the Interface class and handle the lifecycle of an array, including coercion, validation, and serialization.

    Discovery

    Interfaces are automatically discovered via the Interface.interfaces() method, which returns all registered subclasses. To use a custom interface, you must ensure it is defined or imported before you instantiate a Pydantic model that requires it.

    Matching Logic

    When a Pydantic model is instantiated, Interface.match is called to find the appropriate interface for the input data:

    1. Interface.check: Each interface implements a check class method. This method receives the input and returns whether the interface can handle it (e.g., checking if a path points to a specific file type). This should be fast as it is called frequently.
    2. Interface.enabled: Determines if the interface is available for use, typically by checking if its required dependencies are installed in the environment.
    3. NumpyInterface Special Case: The NumpyInterface is only checked if no other interface matches. This prevents expensive operations (like casting a lazy-loaded array to a full numpy.ndarray in memory) from occurring prematurely.

  4. Interface support for Datetimes, Objects, and Strings

    main

    Different array interfaces have varying levels of support for specific dtypes:

    • Datetimes: Supported by every interface except VideoInterface. Note: HDF5 loses timezone information; timestamps should be re-encoded to UTC before saving/loading.
    • Objects: Supported by all interfaces except VideoInterface, HDF5Interface, and ZarrInterface. For the numpy interface, validation only checks the first item in the array for object identity to maintain performance.
    • Strings: Supported by all interfaces except VideoInterface.

  5. Array specification syntax with nptyping

    main

    Numpydantic adopts the nptyping syntax for specifying array shapes and dtypes within NDArray annotations. This allows for granular type annotations.

    While numpydantic is moving towards a native implementation in future versions, you can currently use existing nptyping annotations as-is. Numpydantic also provides minor modifications to the numpydantic.dtype specification, such as allowing built-in Python types like int and float directly.

  6. Mypy Plugin: Pydantic Model Integration

    main

    When using Numpydantic within a Pydantic model, the plugin allows Pydantic to coerce convenience inputs (like file paths or strings) into actual arrays at runtime.

    To ensure static type checking works correctly for these fields, you should set init_typed = true in your [tool.pydantic-mypy] configuration.

    Note: Shape and dtype checking is intentionally unsupported for non-array inputs (like paths) within Pydantic models to avoid impossible tasks like opening on-disk stores during type checking.

  7. How NDArray and interfaces work together

    main

    Numpydantic uses numpydantic.NDArray as an abstract specification for arrays. Instead of being a standalone array library, it acts as a bridge between Pydantic schemas and existing array libraries (like NumPy, Dask, etc.).

    To handle different array behaviors (e.g., HDF5 files requiring careful closing or video files not supporting standard slicing), numpydantic uses Interface classes.

    When you use an NDArray specification, numpydantic selects an appropriate interface to validate and interact with the data. The interface may return:

    1. The original array itself.
    2. A transparent proxy class (e.g., numpydantic.interface.hdf5.H5Proxy) if the native format requires it to support NumPy-like operations.

    To prevent IDEs and static type checkers (like Pyright or Mypy) from complaining when a proxy is returned instead of the raw array, numpydantic automatically generates a .pyi stub file on import. This stub declares NDArray as a Union of all possible .Interface.return_types.

  8. Understand numpydantic versioning and API stability

    main

    numpydantic uses a modified version of Semantic Versioning 2.0.0 during its current 2.*.* phase.

    Versioning Behavior (Current 2.*.* phase)

    • Breaking changes: Indicated by an advance in the MINOR version (instead of MAJOR).
    • Backwards-compatible bugfixes and new functionality: Indicated by a PATCH release (instead of MINOR and PATCH).
    • Recommendation: Set an upper bound on your version dependencies until the 2.*.* phase is complete.

    API Stability Tiers

    • Public API: Only the .NDArray and .Shape classes (and their associated validation functionality) are considered stable. The removal of string-form shape specifications is planned for v3.*.*.
    • Development API: Only the .Interface class and its subclasses, along with the Public API, are considered stable. This is the primary point for external development.
    • Other components: Everything else (including import structures or private members) is subject to change without notice.

    Deprecation Policy

    • Public API breaking changes: Signaled by a major version's worth of deprecation warnings.
    • Development API breaking changes: Signaled by a minor version's worth of deprecation warnings.

  9. Mypy Plugin: Shape and Dtype Checking

    main

    The Mypy plugin enables several advanced type-checking capabilities:

    1. Shape Compatibility: Checks .NDArray annotations against specific shapes, ranges, wildcards, and ellipses.
      • Scalars: Must match exactly.
      • Ranges: Works on the left-hand side with a right-hand side scalar.
      • Wildcards (*): Specifies a dimension must exist but can be any size.
      • Ellipses (...): Specifies any number of additional dimensions may be included.
    2. Dtype Checking: Validates that array dtypes match the annotation.
    3. Constructor Inference: Infers shapes and dtypes from supported constructors (like numpy.ones) when using integer literals.
      • Note: Dask constructor inference is currently not supported due to its dynamic construction pattern.

  10. Use Shape forms for typechecking

    main

    Numpydantic supports several ways to express shapes, but they differ in their compatibility with static type checkers (like Mypy).

    1. Tuple Form (Preferred)

    This is the recommended form for versions >= v2.0. While the technically correct form uses typing.Literal, the numpydantic Mypy plugin allows a more readable version.

    Recommended (with Mypy plugin):

    NDArray[Shape[1, "2-3", "*", "..."]]

    Strict/Literal form (no plugin):

    from typing import Literal as L
NDArray[Shape[L[1], L["2-3"], L["*"], L["..."]]]

    2. String Form (Discouraged)

    Inherited from nptyping. This is syntactically invalid to the Python type system and will be deprecated in v2.0 and removed in v3.0.

    NDArray[Shape["1, 2-3, *, ..."]]

    3. Functional Form

    Use this within NDArraySchema or when other forms fail static type checking.

    Annotated[np.ndarray, NDArraySchema(Shape(1, "2-3", "*", "..."))]
    NDArray[Shape[1, "2-3", "*", "..."]]

  11. Understand Numpydantic dtype coercion behavior

    main

    Numpydantic follows a design principle of balancing usability with the minimization of surprise. Unlike standard Pydantic, which heavily coerces scalar values, Numpydantic avoids expensive array operations like reshaping or changing dtypes to prevent data corruption.

    Numpydantic WILL:

    • Coerce non-array values (scalars or Python sequences) into numpy arrays.
    • Coerce recognized Interface.input_types to the matching Interface.return_type (e.g., converting a pathlib.Path to a VideoProxy).
    • Coerce lists or arrays of dicts into annotated pydantic.BaseModel instances.

    Numpydantic WILL NOT:

    • Attempt to reshape an input array.
    • Attempt to change the dtype of an existing array.

  12. Enable static array analysis with the Mypy plugin

    main

    The Numpydantic Mypy plugin enables static analysis of array shapes and dtypes. Without the plugin, array constructors like np.zeros are often seen as Any. With the plugin, Mypy can track specific dimensions (as Literal types) and catch shape or dtype mismatches during static type checking.

    # With the numpydantic mypy plugin enabled:

# Mypy will reveal specific Literal types for shape
reveal_type(np.zeros((3, 4), dtype=np.uint8)) 
# Revealed type is "numpy.ndarray[tuple[Literal[3], Literal[4]], numpy.uint8]"

# This will trigger a static type error if the shape doesn't match
def analyze_grayscale(frame: NDArray[Shape[1080, 1920], np.uint8]):
    ...

# Error: Argument "frame" has incompatible type ... expected shape [1080, 1920]
analyze_grayscale(np.zeros((1080, 1920, 3), dtype=np.uint8))