DOCSEARCHbyAlgolia
Submit Docs

zarr-python

repository·main·Indexed 24 days ago

https://github.com/zarr-developers/zarr-python

A Python implementation of compressed, chunked, N-dimensional arrays designed for high-performance parallel computing. It supports flexible storage backends, NumPy dtypes, and NumCodecs for compression. The project includes specialized packages such as zarr-indexing for coordinate transforms and zarr-metadata for Zarr v2 and v3 metadata validation and modeling.

Tokens
53K
Snippets
108
Records
331
Agent score
84%

What's inside zarr

  1. What is zarr-metadata?

    main

    zarr-metadata provides Python types, models, and validators for Zarr v2 and v3 metadata. It consists of three main components:

    1. Typed JSON shapes: TypedDict definitions and Literal aliases representing the JSON documents specified by the Zarr v2 and v3 specifications, including zarr-extensions and consolidated metadata.
    2. Document models (zarr_metadata.model): Canonical frozen-dataclass models of whole metadata documents. These include structural validators, loc-aware parsers, and store-key (de)serialization. Models produced via to_json are decoupled from the original model's mutable state.
    3. Optional Pydantic integration (zarr_metadata.pydantic): Requires Pydantic 2.13 or newer. Each model can be used as a Pydantic field type that validates raw documents using the same strict parser used by the document models.

  2. Overview of Zarr features

    main

    Zarr is a Python package that implements compressed, chunked, N-dimensional arrays optimized for parallel computing.

    Key capabilities include:

    • Array Creation: Create N-dimensional arrays using any NumPy dtype.
    • Chunking: Chunk arrays along any dimension for optimized performance.
    • Compression/Filtering: Use any NumCodecs codec to compress or filter chunks.
    • Flexible Storage: Store arrays in various backends including memory, local disk, zip files, or S3.
    • Concurrency: Read and write arrays concurrently using multiple threads or processes.
    • Hierarchical Organization: Organize multiple arrays into hierarchies using groups.

  3. Overview of Zarr-Python features

    main

    Zarr-Python is a library for reading and writing Zarr groups and arrays. It provides:

    • Specification support: Works with both Zarr format 2 and 3.
    • NumPy-like semantics: Create and read N-dimensional arrays using familiar interfaces.
    • Flexible storage: Supports local, cloud, and in-memory stores.
    • High performance: Supports asynchronous I/O and multi-threading for fast I/O.
    • Extensibility: Allows for user-defined codecs and stores.

  4. Extend Zarr-Python using abstract base classes

    main

    The zarr.abc module provides a set of abstract base classes (ABCs) designed for developers who need to extend the core functionality of Zarr-Python. Depending on your extension goal, you should implement one of the following interfaces:

    • zarr.abc.buffer: Use this to provide access to underlying memory via Python buffers.
    • zarr.abc.codec: Use this to implement custom Zarr codecs according to the Zarr specification.
    • zarr.abc.metadata: Use this to create custom metadata classes that remain compatible with the Zarr API.
    • zarr.abc.numcodec: Use these protocols and classes when modeling codec interfaces intended for use by numcodecs.
    • zarr.abc.store: Use this to implement custom Zarr stores, managing the low-level getting and setting of bytes within the store.

  5. Understand the zarr-metadata package structure

    main

    The zarr-metadata package is organized to mirror the Zarr specifications. It provides models, validators, and type definitions for different Zarr versions:

    • zarr_metadata.model: Contains frozen-dataclass document models, structural validators, loc-aware parsers, and the UNSET sentinel.
    • zarr_metadata.pydantic: Provides optional Pydantic field types that can be used over the models.
    • zarr_metadata.v2: Provides TypedDict shapes for Zarr v2 documents, including .zarray, .zgroup, .zattrs, and .zmetadata.
    • zarr_metadata.v3: Provides TypedDict shapes for Zarr v3 documents. This is further divided into subpackages for:

    Note on Imports: All public names are re-exported at the top level. For example, from zarr_metadata import ZarrV3ArrayMetadataJSON is equivalent to from zarr_metadata.v3.array import ZarrV3ArrayMetadataJSON.

  7. Core Zarr API: Essential Classes and Functions

    main

    The core Zarr API revolves around managing N-dimensional data through hierarchical structures.

    • Array: The primary class representing N-dimensional data.
    • Group: Used for hierarchical organization, allowing you to nest arrays and subgroups.
    • Creation: Use create_array() to initialize new arrays and create_group() to initialize new groups.
    • Opening: Use open(), open_array(), or open_group() to access existing Zarr stores, arrays, or groups.

  9. Core Zarr API: Storage and Compression

    main

    Zarr's flexibility comes from its decoupled storage and compression layers:

    • Codecs: Handle compression and filtering of data chunks.
    • Storage: Provides the backend implementations (e.g., filesystem, cloud storage) and interfaces for where data is physically kept.
    • Registry: Manages the registration of codecs and storage backends.

  10. Goals of the Unified Chunk Grid implementation

    main

    The implementation of the Unified Chunk Grid aims to achieve the following:

    1. Spec Compliance: Conforms to the rectilinear chunk grid spec.
    2. API Stability: Minimizes changes to the public API. Users creating regular arrays should experience no difference, as rectilinear support is additive.
    3. Backwards Compatibility: Existing code using .chunks, isinstance checks, or importing RegularChunkGrid/RectilinearChunkGrid from zarr.core.chunk_grids will continue to work.
    4. Future-Proofing: Internal architecture allows for refactoring (like metadata/array separation) without breaking the public API.
    5. Minimal Downstream Impact: Designed to require minimal updates from libraries like xarray, VirtualiZarr, Icechunk, and Cubed.
    6. Useful Public API: Provides practical methods such as read_chunk_sizes/write_chunk_sizes, ChunkGrid.__getitem__, and is_regular to solve real-world problems rather than just exposing internals.

  11. Explore advanced Zarr-Python topics

    main

    For specialized use cases, explore these advanced topics:

    • Data Types: Learn about supported and extensible data types.
    • Performance: Techniques to optimize for speed and efficiency.
    • GPU: How to leverage GPU acceleration.
    • Extending: How to extend Zarr-Python functionality with custom code.
    • Consolidated Metadata: Advanced management of metadata.
    • Experimental Features: Preview features that are subject to change.