DOCSEARCHbyAlgolia
Submit Docs

HDF5 Library Documentation

repository·develop·Indexed 21 days ago

https://github.com/hdfgroup/hdf5

A high-performance library and data model for managing complex, large-scale data. This documentation covers the HDF5 C and Fortran APIs, CMake configuration for builds, and development tools including sanitizers, code coverage, AFL++ fuzzing, static analysis (clang-tidy, cppcheck, IWYU), and automated code formatting.

Tokens
46.7K
Snippets
90
Records
173
Agent score
75%

What's inside HDF5

  1. Overview of scripts in the `bin` directory

    develop
    The bin directory contains various utility scripts used for development, build processes, and code maintenance within the HDF5 project. These scripts range from API checkers and source formatters to code generators for headers and parsers.

  2. Overview of the HDF5 Fortran Library Modules

    develop

    The HDF5 Fortran library is organized into several functional modules that allow you to manage different aspects of HDF5 data structures and file operations. These modules include:

    • Files (H5F): Manage HDF5 files.
    • Groups (H5G): Manage HDF5 groups (containers for datasets and other groups).
    • Datasets (H5D): Manage datasets, including data transfer between memory and disk and dataset properties.
    • Attributes (H5A): Manage small metadata objects describing primary data objects.
    • Dataspaces (H5S): Describe the shape/dimensions of datasets in memory or files.
    • Datatypes (H5T): Describe the element types of datasets and attributes.
    • Property Lists (H5P): Configure the behavior of HDF5 API functions.
    • Links (H5L): Manage HDF5 links and link types.
    • Identifiers (H5I): Manage identifiers defined by the HDF5 library.
    • Objects (H5O): Manage HDF5 objects (groups, datasets, datatype objects).
    • References (H5R): Manage references to HDF5 objects, attributes, and dataset regions.
    • Error Handling (H5E): Handle library error reporting.
    • Filters (H5Z): Manage user-defined filters.
    • Library General (H5): Manage the lifecycle of HDF5 library instances.

    High-Level Interfaces:

    • High Level Lite (H5LT): Simplified functions for creating and manipulating datasets and attributes.
    • High Level Image (H5IM): Specialized for datasets intended to be interpreted as images.
    • High Level Table (H5TB): Specialized for datasets intended to be interpreted as tables.
    • High Level Dimension Scale (H5DS): For datasets associated with the dimensions of another dataset.

  3. Overview of the HDF5 Java Interface packages

    develop

    The HDF5 Java interface is organized into several specialized packages that map to the core HDF5 C library functionalities. Developers should use these packages based on the specific HDF5 concept they are interacting with:

    • Core Library: JH5 provides the primary interface.
    • Data Management:
      • JH5D for Dataset APIs.
      • JH5S for Dataspace APIs.
      • JH5T for Datatype APIs.
      • JH5G for Group APIs.
      • JH5O for Object APIs.
    • Metadata and Organization:
      • JH5A for Attribute APIs.
      • JH5F for File APIs.
      • JH5L for Links APIs.
      • JH5I for Identifier APIs.
    • Configuration and Extensions:
      • JH5P for Property List APIs.
      • JH5Z for Filter APIs.
      • JH5PL for Plugin APIs.
      • JH5VL for VOL (Virtual Object Layer) connector APIs.
      • JH5R for Reference APIs.
    • Utilities and Constants:
      • HDF5CONST: Contains C constants and enumerated types.
      • HDFNATIVE: Handles native methods for converting between numbers and byte arrays.
      • HDFARRAY: Manages multidimensional arrays for HDF5.
    • Error Handling:
      • ERRORS: Uses the HDF5Exception class to return errors from the interface.

  4. Core HDF5 Library API Modules

    develop

    The HDF5 Core Library is organized into several functional modules that allow you to manage the lifecycle of files, data, and metadata. Key modules include:

    • Files (H5F): For managing HDF5 files.
    • Groups (H5G): For managing HDF5 groups (containers for datasets and other groups).
    • Datasets (H5D): For managing datasets, including data transfer between memory and disk and dataset properties.
    • Dataspaces (H5S): For describing the shape/dimensions of datasets in memory or on disk.
    • Datatypes (H5T): For describing the element types within datasets and attributes.
    • Attributes (H5A): For managing small metadata objects attached to primary data objects.
    • Links (H5L): For managing HDF5 links and link types.
    • References (H5R): For managing references to HDF5 objects, attributes, and dataset regions.
    • Objects (H5O): For managing general HDF5 objects (groups, datasets, etc.).
    • Identifiers (H5I): For managing library-defined identifiers.
    • Property Lists (H5P): The primary mechanism for configuring the behavior of HDF5 API functions.
    • Filters (H5Z): For managing user-defined filters (e.g., compression).
    • Error Handling (H5E): For library error reporting.
    • Library General (H5): For managing the lifecycle of HDF5 library instances.
    • VOL Connector (H5VL): For managing HDF5 Virtual Object Layer (VOL) connector plugins.
    • Event Set (H5ES): For managing the HDF5 event set lifecycle, used with VOL connectors to enable asynchronous features.
    • Dynamically-loaded Plugins (H5PL): For managing the loading behavior of HDF5 plugins.

  5. Use the HDF5 High-level Library for common operations

    develop

    The HDF5 High-level Library provides sets of convenience and standard-use APIs designed to simplify common HDF5 tasks. Instead of using low-level primitives, you can use these specialized modules to handle specific data structures and patterns efficiently.

    Key high-level modules include:

    • H5LT: Functions to simplify creating and manipulating datasets, attributes, and other HDF5 features.
    • H5IM: Specialized for creating and manipulating HDF5 datasets intended to be interpreted as images.
    • H5TB: Specialized for creating and manipulating HDF5 datasets intended to be interpreted as tables.
    • H5PT: Supports append- and read-only operations on table data.
    • H5DS: Manages datasets that are associated with the dimensions of another HDF5 dataset.
    • H5DO: Provides mechanisms to bypass default HDF5 behavior to optimize for specific use cases.
    • H5LR: Provides extensions for HDF5 operations.

  6. How Parallel and Threadsafe modes affect HDF5 builds

    develop

    HDF5 supports two specialized modes that are mutually exclusive and have restrictions on which interfaces can be built:

    Parallel Mode

    Enables MPI and MPI-IO for distributed multi-processor systems.

    • Requires: HDF5_ENABLE_PARALLEL:BOOL=ON.
    • Incompatible with: HDF5_ENABLE_THREADSAFE, HDF5_BUILD_CPP_LIB, and HDF5_BUILD_JAVA.

    Threadsafe Mode

    Enables large-scale thread safety.

    • Requires: HDF5_ENABLE_THREADSAFE:BOOL=ON.
    • Incompatible with: HDF5_BUILD_HL_LIB, HDF5_BUILD_FORTRAN, HDF5_BUILD_CPP_LIB, and HDF5_BUILD_JAVA.

    Note: To bypass these restrictions, you must set HDF5_ALLOW_UNSUPPORTED to ON during configuration.

  7. Handle unsupported HDF5 feature combinations

    develop

    Some HDF5 features are incompatible and will cause a configuration error unless HDF5_ALLOW_UNSUPPORTED is set to ON.

    Incompatible Combinations

    1. Parallel HDF5 (HDF5_ENABLE_PARALLEL=ON) is incompatible with:

      • HDF5_ENABLE_CONCURRENCY
      • HDF5_ENABLE_THREADSAFE
      • HDF5_BUILD_CPP_LIB

    2. Concurrency/Thread-safety (HDF5_ENABLE_CONCURRENCY or HDF5_ENABLE_THREADSAFE) is incompatible with high-level interfaces because locking is not hoisted into them:

      • HDF5_BUILD_HL_LIB
      • HDF5_BUILD_FORTRAN
      • HDF5_BUILD_JAVA
      • HDF5_BUILD_CPP_LIB

    3. Concurrency vs Thread-safety: HDF5_ENABLE_CONCURRENCY and HDF5_ENABLE_THREADSAFE are mutually exclusive; only one can be enabled.

  8. Revoke HDF5 plugin signatures

    develop

    If a plugin signature is compromised, you can revoke it by adding its identifier to a revocation file within your keystore directory.

    Revocation File Format

    1. Create a file named revoked_signatures.txt in your keystore directory.
    2. Each line must contain the 64-character hex-encoded SHA-256 hash of the raw signature bytes.
    3. Lines starting with # are treated as comments. Empty lines are ignored.

    Note: Do not use the hex-encoded raw signature itself. You must compute the SHA-256 digest of the raw signature bytes first.

    Example revoked_signatures.txt

    # SHA-256 hash of a compromised plugin's signature
a1b2c3d4e5f6...

  9. Manage HDF5 File Format Compatibility

    develop

    HDF5 2.0.0 changes the default file creation behavior. New files now use the 1.8-era format by default for improved performance and space efficiency.

    • Backward Compatibility: Files created with 2.0.0 defaults are readable by HDF5 versions as old as 1.8.0.
    • Forward Compatibility: Newer libraries can read older files. However, if you use new features (like the complex number datatype class), older libraries will not be able to read them.
    • Supporting Older Readers (< 1.8.0): If your application must produce files readable by HDF5 versions older than 1.8.0, you must explicitly set the version bounds using H5Pset_libver_bounds() during file creation.
    // Example: Explicitly setting version bounds to support older readers
H5Pset_libver_bounds(plist, H5F_VERSION_1_6, H5F_VERSION_1_8);

  10. Cross-compiling HDF5 with CMake

    develop

    When cross-compiling, CMake cannot automatically detect the target platform or find libraries in default system directories. You must provide a toolchain file using the CMAKE_TOOLCHAIN_FILE variable.

    Key Toolchain Variables:

    • CMAKE_C_COMPILER / CMAKE_CXX_COMPILER: Paths to the target compilers.
    • CMAKE_AR, CMAKE_LINKER, CMAKE_OBJCOPY, CMAKE_RANLIB: Paths to other required toolchain binaries.
    • CMAKE_SYSTEM_NAME: The name of the target platform (e.g., Linux, Windows, Darwin).
    • CMAKE_FIND_ROOT_PATH: Path to the target's root filesystem (sysroot) containing pre-compiled libraries/headers.

    Best Practice: Put these settings in a <toolchain_name>.cmake file and invoke CMake with: cmake -DCMAKE_TOOLCHAIN_FILE=path/to/file ...

  11. Understand the relationship between C and Fortran HDF5 APIs

    develop

    The Fortran APIs are organized into modules that run parallel to the HDF5 C interfaces.

    • Module Files: Each module is contained in a file named H5*ff.F90 (e.g., H5Fff.F90 for File APIs).
    • C Stubs: Corresponding C stubs are located in H5*f.c files (e.g., H5Ff.c).
    • Module Contents: Each module includes Fortran definitions for constants, subroutine interfaces, and the subroutines themselves.

  12. Understand the CMake installation directory structure

    develop

    When installing HDF5 via CMake, the installation produces the following standard directory structure:

    • bin/: Contains build scripts and tools (e.g., h5c++, h5cc, h5hlc++, h5hlcc). CMake also creates dynamic versions of these tools with the -shared suffix.
    • include/: Contains header files and Fortran module files. If HDF5_INSTALL_MOD_FORTRAN is used, Fortran module files are placed directly in include/ instead of separate subfolders.
    • lib/: Contains library files and a pkgconfig/ subfolder containing hdf5*.pc files. CMake separates C interface code from Fortran code by creating C-stub libraries for each Fortran library.
    • share/: Contains documentation, including the LICENSE and CHANGELOG.md files.