kerchunk

Kerchunk is a library designed for cloud-friendly, efficient access to archival data stored in chunked, compressed formats (such as NetCDF, HDF5, GRIB, TIFF, and FITS).

It works by extracting metadata—including byte ranges and compression information—and storing it in a separate object. This allows you to create virtual datasets that aggregate multiple source files into a single logical view.

Key Use Cases:

• Cloud-native access to legacy formats: Access data stored in object storage (S3, GCS, ABFS, etc.) without needing to copy or translate the original files.
• Metadata consolidation: Understand complex, many-file datasets through a single metadata read.
• Serverless data processing: Avoid the need for specific drivers (like h5py) by using Kerchunk as a gateway to in-situ data access.
• High-performance parallel reads: Enable asynchronous, concurrent fetching of data chunks and parallel access via libraries like Zarr without locking issues.

While kerchunk documentation primarily focuses on zarr, the project is designed to support other use cases where accessing files in specific directory structures or reading binary pieces of large files at known offsets is required.

Supported or planned non-zarr use cases include:

• .tar.zstd: Enabling parallel/random access to compressed archives by indexing file offsets within the TAR format and utilizing Zstd's block-wise decompression capabilities. This allows pulling only necessary data parts without downloading and unpacking whole files.
• .csv/.json: Enabling parallel/distributed processing of text files that contain embedded newlines (e.g., within quoted strings). By scanning the file once to identify valid record terminators, kerchunk can provide safe random access offsets.
• parquet/orc/feather:
  • Parquet: Using the virtual file system to perform 'smart concatenation' by assigning partition values (e.g., month=may/) to files even if they weren't originally part of the same dataset. This allows for efficient data partitioning without moving or copying files.
  • Feather: Constructing logical Feather files from disparate buffer pieces and serialized metadata to be compatible with the pyarrow API.

Once a k_index has been created for the desired duration, you can use the DataTree model from the xarray-datatree library to view specific variables or the entire aggregation. This allows you to navigate the hierarchical structure of the aggregated GRIB data (e.g., by variable name, step type, or level).

# Example representation of a DataTree for GEFS model aggregation
DataTree('None', parent=None)
├── DataTree('prmsl')
│   │   Dimensions:  ()
│   │   Data variables: 
│   │       *empty*
│   │   Attributes:
│   │       name:     Pressure reduced to MSL
│   └── DataTree('instant')
│       │   Dimensions:  ()
│       │   Data variables: 
│       │       *empty*
│       │   Attributes:
│       │       stepType:  instant
│       └── DataTree('meanSea')
│               Dimensions:     (latitude: 181, longitude: 360, time: 1, step: 1, model_horizons: 1, valid_times: 237)
│               Coordinates:
│                 * latitude    (latitude) float64 ...
│                 * longitude   (longitude) float64 ...
│                 * meanSea     float64 ...
│                 * number      (time, step) int64 ...
│                 * step        (model_horizons, valid_times) timedelta64[ns] ...
│                 * time        (model_horizons, valid_times) datetime64[ns] ...
│                 * valid_time  (model_horizons, valid_times) datetime64[ns] ...
│               Dimensions without coordinates: model_horizons, valid_times
│               Data variables:
│                   prmsl       (model_horizons, valid_times, latitude, longitude) float64 ...
│               Attributes:
│                   typeOfLevel:  meanSea
└── DataTree('ulwrf')
    │   Dimensions:  ()
    │   Data variables: 
    │       *empty*
    │   Attributes:
    │       name:     Upward long-wave radiation flux
    └── DataTree('avg')
        │   Dimensions:  ()
        │   Data variables: 
        │       *empty*
        │   Attributes:
        │       stepType:  avg
        └── DataTree('nominalTop')
                Dimensions:     (latitude: 181, longitude: 360, time: 1, step: 1, model_horizons: 1, valid_times: 237)
                Coordinates:
                  * latitude    (latitude) float64 ...
                  * longitude   (longitude) float64 ...
                  * nominalTop  float64 ...
                  * number      (time, step) int64 ...
                  * step        (model_horizons, valid_times) timedelta64[ns] ...
                  * time        (model_horizons, valid_times) datetime64[ns] ...
                  * valid_time  (model_horizons, valid_times) datetime64[ns] ...
                Dimensions without coordinates: model_horizons, valid_times
                Data variables:
                    ulwrf       (model_horizons, valid_times, latitude, longitude) float64 ...
                Attributes:
                    typeOfLevel:  nominalTop

If the dimension you want to concatenate along does not exist in the source datasets, or if you are combining datasets from an ensemble, use the coo_map argument to create a new dimension.

You can provide:

1. Literal values: A list of values to populate the new dimension.
2. Regex functions: A compiled regex that extracts values from the input file URLs to populate the dimension.
3. Custom functions: A function with the signature (index, fs, var, fn) -> value to generate complex types like datetime.datetime.

# Example 1: Using literal values
mzz = MultiZarrToZarr(
    json_list,
    remote_protocol='s3',
    remote_options={'anon':True},
    coo_map = {'new_dimension': ['a', 'b']},
    concat_dims=['new_dimension'],
    identical_dims = ['lat', 'lon']
)

# Example 2: Using a regex function
import re
ex = re.compile(r'.*(\d+)_air')
mzz = MultiZarrToZarr(
    json_list,
    remote_protocol='s3',
    remote_options={'anon':True},
    coo_map = {'new_dimension': ex},
    concat_dims=['new_dimension'],
    identical_dims = ['lat', 'lon']
)

# Example 3: Using a custom function for datetime extraction
def fn_to_time(index, fs, var, fn):
    import re
    import datetime
    subst = re.search(r"\d{12}", fn)[0]
    return datetime.datetime.strptime(subst, '%Y%m%d%H%M')

mzz = MultiZarrToZarr(
    sorted(glob.iglob(r'*.json')),
    remote_protocol='file',
    coo_map={'time': fn_to_time},
    coo_dtypes={'time': np.dtype('M8[s]')},
    concat_dims=['time'],
    identical_dims=['lat', 'lon'],
)

Once a report is submitted to community@anaconda.com, the core team follows this process:

1. Acknowledgment: You will receive an email acknowledging receipt of your complaint.
2. Review: The core team meets to determine the facts, whether a violation occurred, the identity of the actor, and if there is an ongoing threat to safety.
3. Conflict Management: If a core team member is involved in the incident or has a conflict of interest, they are excluded from discussions and denied access to confidential details.
4. Resolution: The team decides on a response, which may include:
   • No action (if no violation is found).
   • Private or public reprimand.
   • Imposed vacation.
   • Temporary or permanent ban from project spaces (e.g., GitHub repositories).
   • Request for a public or private apology.
5. Communication: The team will respond to the reporter within one week with either a resolution or an explanation of the delay. Once a final action is determined, the reporter will be notified of the outcome.

For GRIB files, you can significantly speed up reference aggregation by using accompanying .idx files. This method avoids scanning the entire archive by using the metadata contained in the index files to build the reference.

Requirements and Restrictions:

• GRIB files must be paired with their corresponding .idx files.
• The .idx files must be of text type.
• This method is specialized for time-series data where GRIB files have an identical structure.
• Each horizon (forecast time) must be indexed separately.

The Three-Step Workflow:

1. Extract Metadata: Extract and persist metadata from a few arbitrary GRIB files for a specific product (e.g., HRRR, GEFS, GFS).
2. Build Index Table: Use the metadata mapping to build an index table of every GRIB message extracted from the .idx files.
3. Combine: Combine the index data with the metadata to build any FMRC slice (Horizon, RunTime, ValidTime, BestAvailable).