DOCSEARCHbyAlgolia
Submit Docs

VirtualiZarr

repository·main·Indexed 43 days ago

https://github.com/zarr-developers/virtualizarr

A tool for cloud-optimizing scientific datasets by creating virtual Zarr datacubes from archival formats like netCDF4 and HDF5 without data duplication. It provides cloud-optimized access via Xarray and Zarr APIs, supporting Zarr v3, ManifestStore abstraction, and integration with Icechunk and Kerchunk for committing virtual references.

Tokens
48.4K
Snippets
141
Records
192
Agent score
54%

What's inside VirtualiZarr

  1. What is VirtualiZarr?

    main

    VirtualiZarr is a tool for cloud-optimizing scientific datasets by creating "Virtual" Zarr datacubes. It allows you to access archival, non-cloud-optimized file formats (like netCDF4 and HDF5) as if they were in the performant Zarr format, without actually duplicating or moving the underlying data.

    Key capabilities include:

    • Creating virtual references to specific bytes within archival files using open_virtual_dataset.
    • Combining multiple archival files into a single large datacube using Xarray functions like xarray.concat.
    • Accessing data via the zarr-python API through a ManifestStore.
    • Committing virtual references to storage using either the Kerchunk specification or the Icechunk transactional storage engine.
    • Using xarray.open_zarr to consume the resulting virtual datacube as a single Zarr-compatible store.

  2. What is a Chunk Manifest?

    main

    A Chunk Manifest is the core data structure in VirtualiZarr that maps Zarr-style chunk keys (e.g., "0.0.0") to specific byte ranges within external files. Instead of storing actual compressed binary files, a manifest stores a list of path, offset, and length attributes. This allows a single virtual Zarr array to be composed of chunks spread across any number of files and locations (local or remote, like S3).

    # Example manifest structure
{
    "0.0.0": {"path": "s3://bucket/foo.nc", "offset": 100, "length": 100},
    "0.0.1": {"path": "s3://bucket/foo.nc", "offset": 200, "length": 100},
    "0.1.0": {"path": "s3://bucket/foo.nc", "offset": 300, "length": 100},
    "0.1.1": {"path": "s3://bucket/foo.nc", "offset": 400, "length": 100},
}

  3. Address inhomogeneous codecs by splitting arrays

    main

    Zarr arrays require all chunks to use identical codecs (e.g., compression settings). If your source archive uses different codecs for different chunks (due to changing compression standards or library defaults), they cannot be virtualized into a single logical array.

    Current Workaround: Split the data into multiple arrays (e.g., grouping by 'era') where the codec remains consistent within each group. This allows you to manage different compression regimes as separate virtual datasets.

  4. Scalability of Kerchunk reference formats

    main

    When persisting virtual references using Kerchunk, the choice of format significantly impacts scalability:

    • In-memory dict and JSON: These are extremely inefficient for large datasets. Converting a virtual dataset that fits in memory to these formats can result in massive memory consumption or disk usage. Not recommended for large datasets.
    • Parquet: This is the most scalable Kerchunk format. To optimize performance, experiment with the record_size and categorical_threshold arguments in the VirtualiZarrDatasetAccessor.to_kerchunk method.

  5. Understand the `ManifestArray` class

    main

    A ManifestArray is a virtualized representation of a single Zarr array. It does not contain actual data in memory; instead, it stores array-level attributes (like shape, dtype, and metadata) alongside a ChunkManifest that points to the location of the constituent chunk data on disk.

    Because it implements array-like methods, it acts as a "duck array," allowing libraries like NumPy or Xarray to treat it as a multidimensional array without special handling.

    Key constraints:

    • No direct loading: You cannot call .values on a ManifestArray to get a NumPy array; doing so raises a NotImplementedError. The purpose is to manipulate references, not load data.
    • Indexing: You can index into a ManifestArray only if the selection aligns with chunk boundaries. Slicing through the interior of a chunk raises a SubChunkIndexingError.
    • Concatenation: Multiple ManifestArray objects can be concatenated (e.g., via np.concatenate) by merging their chunk manifests. This allows combining data from multiple files into a single virtual Zarr store.
    • Codec Restriction: You cannot concatenate arrays that use different compression codecs.
    # Example of a ManifestArray representation
# ManifestArray<shape=(2920, 25, 53), dtype=int16, chunks=(2920, 25, 53)>

# Concatenating ManifestArrays using numpy
import numpy as np
concatenated = np.concatenate([marr, marr], axis=0)
# Resulting shape: (5840, 25, 53)

  7. Parse sidecar files with custom parsers

    main

    Custom parsers can be designed to handle multiple files or sidecar files. You can achieve this by:

    • Passing a glob string to the parser to find expected file patterns.
    • Using parser-specific keyword arguments to pass additional context.

    This pattern is useful for formats that use an external 'index' file that contains some, but not all, the information required to construct a ManifestStore.

  8. Requirements for virtualizing file formats

    main

    To be 'virtualizable' into a Zarr-compatible format, a file format must meet two primary requirements for efficient cloud access via HTTP range requests:

    1. Contiguous Chunks: Each chunk of array data (compressed block) must be accessible via a single targeted HTTP range request without pulling significant extraneous bytes. The file format must arrange data such that each chunk on disk corresponds to a single Zarr chunk. (Note: formats like gzipped NetCDF4 are not virtualizable because gzip scrambles byte order).
    2. Efficient Decoding: The bytes must be decodable by a relatively cheap, self-contained, per-chunk transform (a Zarr codec). Text-based formats like CSV are generally not suitable because they require expensive decoding steps.

    Cloud-native formats like Zarr are inherently virtualizable. Tabular formats like Parquet or Iceberg are cloud-optimized but are generally not virtualizable as Zarr due to their internal encodings (e.g., dictionary-encoded pages).

  9. Handle fill values correctly in custom parsers

    main

    When implementing a custom parser, you must distinguish between two different 'fill value' concepts to ensure compatibility with readers like xarray:

    1. Zarr fill_value: The default value returned for uninitialized or missing chunks. This is set via the fill_value parameter in ArrayV3Metadata.
    2. CF _FillValue: A sentinel value (attribute) used by CF-aware readers to mask individual data points within a chunk. This is stored as a Zarr attribute.

    Mapping source formats:

    • HDF5: Map dataset.fillvalue (storage-level) to Zarr fill_value. Carry the CF _FillValue attribute through separately.
    • DMR++: Map fillValue (chunk-level) to Zarr fill_value. Carry CF _FillValue as an attribute.
    • FITS: Map the BLANK keyword to either fill_value or _FillValue depending on whether it represents uninitialized storage or a data-level sentinel.
    • NetCDF-3: If no explicit _FillValue is present, apply the NetCDF-3 default for the variable's type (e.g., 9.9692e+36 for floats).

    Important for Packed Data: If using scale_factor/add_offset, the _FillValue must be in the packed (encoded) domain.

  10. Estimate memory requirements for virtualization

    main

    Scaling VirtualiZarr requires monitoring memory at three stages:

    1. Reference Generation: Each worker running open_virtual_dataset must have enough memory to read the file (depends on the parser and caching).
    2. Reference Combination: The machine calling open_virtual_mfdataset must hold all virtual references in memory. To estimate this, use the .nbytes accessor on a single virtual dataset and multiply by the total number of files: estimated_mem = single_vds.nbytes * num_files. Note: Use the VirtualiZarr .nbytes accessor, not the standard Xarray .nbytes (which returns the size of the full data).
    3. Reference Writing: Converting and writing the combined references to a persistent format may have different memory overheads.

  11. How VirtualiZarr and Kerchunk compare

    main

    VirtualiZarr is inspired by Kerchunk and provides almost all the same features, but it is designed to be "Zarr-native." While Kerchunk focuses on creating reference files, VirtualiZarr provides a more familiar array-like API and integrates more deeply with the Zarr ecosystem.

    In version 2, VirtualiZarr introduced several key architectural improvements over the Kerchunk-based approach:

    • ManifestStore abstraction: Allows loading data without needing to serialize to Kerchunk/Icechunk first.
    • Pluggable Parsers: A system for virtualizing custom file formats and loading data directly from archival formats into Zarr/Xarray.
    • Zarr v3 support.
    • Independent Reference Parsing: Reference parsing that does not rely on Kerchunk under the hood.

  12. Understand data format cloud-optimization tiers

    main

    VirtualiZarr supports various data formats across different levels of cloud optimization. When choosing a format for scientific data processing, consider whether you need to optimize existing legacy formats via virtualization, optimize them during a write operation, or use formats that are natively cloud-optimized or cloud-native.

    Optimization Tiers

    • Non-cloud-optimizable: Formats that cannot be easily optimized for cloud access (e.g., .gz, .bzip2, .zip with deflate).
    • Cloud-optimizable via virtualization: Formats that can be made cloud-efficient using VirtualiZarr's virtualization capabilities (e.g., .tar, .zip without compression, netCDF4/HDF5, GeoTIFF, COG, Native Zarr, Icechunk).
    • Cloud-optimizable upon write: Formats that become cloud-optimized when they are written in a specific way (e.g., netCDF4/HDF5, GeoTIFF, COG, Native Zarr, Icechunk).
    • Cloud-optimized by default: Formats designed to be efficient in the cloud (e.g., Cloud-optimized HDF5, COG, Native Zarr, Icechunk).
    • Cloud-Native (static): Formats that are inherently cloud-native and static (e.g., Native Zarr, Icechunk).
    • Cloud-Native (transactional): Formats that support cloud-native transactional operations (e.g., Icechunk).
    | format                 | Non-cloud-optimizable   | Cloud-optimizable via virtualization | Cloud-optimizable upon write | Cloud-optimized by default | Cloud-Native (static) | Cloud-Native (transactional) |
| ---------------------- | ----------------------- | ------------- | ---------------------------- | -------------------------- | --------------------- | ---------------------------- |
| .tar*                  |                         | x             |                              |                            |                       |                              |
| .gz*                   | x                       |               |                              |                            |                       |                              |
| .bzip2*                | x                       |               |                              |                            |                       |                              |
| .zip (deflate)*        | x                       |               |                              |                            |                       |                              |
| .zip (no compression)* |                         | x             |                              |                            |                       |                              |
| netCDF3                |                         | x             |                              |                            |                       |                              |
| netCDF4 / HDF5         |                         | x             | x                            |                            |                       |                              |
| "Cloud-optimized HDF5" |                         | x             | x                            | x                          |                       |                              |
| HDF4                   |                         | x             |                              |                            |                       |                              |
| GRIB2                  |                         | x             |                              |                            |                       |                              |
| FITS                   |                         | x             |                              |                            |                       |                              |
| TIFF (untiled)         |                         | x             |                              |                            |                       |                              |
| GeoTIFF                |                         | x             | x                            |                            |                       |                              |
| COG                    |                         | x             | x                            | x                          |                       |                              |
| Native Zarr (v2 or v3) |                         | x             | x                            | x                          | x                     |                              |
| Zipped Native Zarr     |                         | x             |                              |                            |                       |                              |
| Icechunk                |                         | x             | x                            | x                          | x                     | x                            |