repository·master·Indexed 19 days ago
https://github.com/juliaio/hdf5.jlA Julia interface to the HDF5 library for efficient storage and access of scientific data. It provides high-level APIs for reading and writing files, support for Julia structs via H5T_COMPOUND, and integration with MPI.jl for parallel I/O. The library supports various compression filters including Blosc, Bzip2, LZ4, and Zstd, as well as virtual datasets and memory-mapped multithreaded reads for contiguous datasets.
H5Zbitshuffle.jl provides an implementation of the bitshuffle filter for use with HDF5.jl in Julia. It implements HDF5 filter ID 32008.
Beyond the standard bitshuffle functionality, it offers optional integrated support for lz4 and zstd (de)compression, allowing for multi-stage data compression pipelines within HDF5 files.
HDF5.jl. It allows you to use LZ4 compression when writing HDF5 files by implementing the HDF5 registered filter ID 32004. This is useful for high-speed compression/decompression workflows within the Julia HDF5 ecosystem.HDF5.jl. It implements HDF5 filter ID 32001. This allows you to use Blosc compression when writing HDF5 files in Julia, which can significantly improve performance and reduce file size for certain data types.H5Zzstd.jl provides the ZStandard (Zstd) filter for HDF5 files in Julia, implementing the HDF5 ZStandard Filter 32015.
Note on Package Status: This is a transitional package. The functionality is now implemented by CodecZstdExt, an extension package for HDF5 that automatically loads when CodecZstd.jl is loaded. Loading H5Zzstd.jl will trigger this extension because it depends on both HDF5.jl and CodecZstd.jl.
bitshuffle_jll_ext package provides the bitshuffle filter implementation for HDF5.jl. It implements HDF5 filter ID 32008 and supports optional integrated lz4 and zstd (de)compression. This allows for efficient data compression within HDF5 files using the bitshuffle algorithm.HDF5.jl package in Julia. It implements the HDF5 registered filter ID 32004. Use this package to enable LZ4 compression when writing HDF5 files, which can offer high-speed compression/decompression for large datasets.CodecZstdExt package provides the Zstd filter implementation for HDF5.jl. It allows you to use ZStandard compression (HDF5 Filter 32015) when writing HDF5 files in Julia. This extension is derived from H5Zzstd.jl and enables high-performance compression for HDF5 datasets.H5Zbzip2.jl package provides an implementation of the Bzip2 compression filter for use with HDF5.jl. It implements the HDF5 registered filter ID 307, allowing you to compress HDF5 datasets using the Bzip2 algorithm within the Julia HDF5 ecosystem.To ensure parallel functionality is available, you must load MPI before HDF5 in your Julia code. This is required because HDF5.jl uses MPI.jl as a dependency that must be initialized first to enable parallel features.
You can verify if your current HDF5 installation supports parallel I/O using HDF5.has_parallel().
using MPI
using HDF5
@assert HDF5.has_parallel()Hyperslabs allow you to select specific, non-contiguous, or regular sub-regions of a dataset for reading or writing. This is essential for efficient I/O when you only need a portion of a large dataset.
Key components and functions:
BlockRange: A type used to define the range of elements in a selection.select_hyperslab!(ds, selection): Selects a specific hyperslab within the dataspace ds using the provided selection.get_regular_hyperslab(ds): Retrieves the current regular hyperslab selection from a dataspace.is_selection_valid(ds): Checks if the current selection on the dataspace is valid.# Example conceptual usage
using HDF5
# Select a regular hyperslab
# (Assuming 'selection' is a valid BlockRange or selection object)
select_hyperslab!(ds, selection)
# Verify the selection
if is_selection_valid(ds)
# Perform I/O on the selected subset
end