DOCSEARCHbyAlgolia
Submit Docs

C-Blosc2 Documentation

repository·main·Indexed 20 days ago

https://github.com/blosc/c-blosc2

A high-performance, compressed, and persistent data store library for C, optimized for binary data such as numerical arrays and tensors. It features an N-Dimensional (B2ND) storage layer and supports various codecs and filters, including NDLZ for 2D arrays, ZFP for lossy floating-point compression, and the NDCELL and NDMEAN multidimensional filters. The library allows for the registration of custom user plugins and provides binary installation via Python wheels.

Tokens
21K
Snippets
38
Records
125
Agent score
69%

What's inside C-Blosc2

  1. Overview of the C-Blosc2 API and compatibility

    main

    The C-Blosc2 API provides a high-performance, compressed, and persistent data store library for C. It is designed with backward compatibility in mind: using the C-Blosc1 API within C-Blosc2 ensures that generated compressed data containers can be read by legacy Blosc1 libraries.

    While maintaining compatibility, the C-Blosc2 API introduces advanced features beyond the original Blosc1 capabilities:

    • 64-bit data containers: Support for larger datasets.
    • Enhanced Filters: A wider range of compression filters.
    • Vector Instruction Support: Improved performance via better hardware acceleration.
    • Flexible Storage: Ability to work with data both in-memory and on-disk using frames.
    • Metadata Support: The ability to attach metainfo to datasets using metalayers.
    • N-Dimensional Storage: Support for multi-dimensional data via Blosc2 NDim.

  2. What is the Blosc2 Sparse Frame (SFrame) format?

    main

    The Sparse Frame (SFrame) format allows for non-contiguous storage of Blosc2 data chunks on disk. Unlike a contiguous frame, an SFrame is represented as a directory containing a frame index file and multiple individual chunk files.

    To create an SFrame, you must:

    1. Set the contiguous flag in the storage struct to false.
    2. Provide a directory name (representing the storage location) in storage.urlpath. It is recommended to use the .b2frame or .b2f extension for these directories.

    SFrame Structure:

    • Frame Index File: Named chunks.b2frame. It contains the metadata and the mapping of the logical chunk order to the physical chunk files. It follows the contiguous frame format but points to multiple files.
    • Chunks: Stored as binary files within the same directory. Each file is named using its index in 8-character, zero-padded, uppercase hexadecimal format with a .chunk extension (e.g., 00000000.chunk, 0000000F.chunk).

    Note: The physical filename of a chunk does not necessarily match its logical order in the data. The actual order is defined by the index chunk, which allows for efficient insertions and reordering without renaming or moving files.

  3. What is Blosc2 NDim (B2ND)?

    main
    The Blosc2 NDim layer (B2ND) allows for efficient creation and reading of n-dimensional datasets. It uses a 2-level partitioning scheme (often called 'pineapple-style' or double partitioning) that enables fine-grained slicing and dicing of arbitrary large, compressed datasets across different axes.

  4. What is a Super-chunk (schunk) in Blosc2

    main

    The super-chunk (or schunk) is the new Blosc 2 container. It acts as a high-level abstraction for managing compressed data. Instead of dealing with individual compressed blocks manually, developers use the schunk API to manage collections of chunks, providing a more structured way to handle compressed data in memory or on disk.

    Key structures involved:

    • blosc2_storage: Represents the storage configuration and metadata.
    • blosc2_schunk: Represents the super-chunk container itself.

  5. Understand the Blosc2 NDim metalayer format

    main
    The Blosc2 NDim (NDim) format is an N-dimensional data store built on top of the standard Blosc2 format. It uses a 'metalayer' to store metadata that describes the N-dimensional structure of the data, such as the shape (dimensions), strides, and data type. This allows the library to treat a collection of Blosc2 chunks as a single, contiguous N-dimensional array, enabling efficient N-dimensional slicing and indexing.

  6. How stale handle revalidation works

    main

    To prevent reading from an unlinked or replaced file (e.g., when a file is rewritten with mode="w"), the cached handle is revalidated using frame_reader_revalidate.

    Before reading header information, the library performs the following check:

    1. fstat the cached file descriptor.
    2. stat the file path.
    3. Compare st_dev and st_ino.

    If a mismatch is detected, the library calls frame_reader_invalidate() and sets force_refresh = true to ensure the new file content is loaded. If the path cannot be stat'd at all, the handle is kept, allowing the library to continue reading the existing inode (standard POSIX behavior).

  7. How Blosc2 contexts enable multithreaded compression and decompression

    main
    In Blosc2, the blosc2_context struct is used to encapsulate compression (blosc2_cparams) and decompression (blosc2_dparams) parameters. By using a context instead of global parameters, you can perform compression and decompression in multithreaded scenarios without needing to acquire a global lock. This allows different threads to operate with their own specific settings independently and safely.

  8. Ensure C-Blosc1 API compatibility

    main
    C-Blosc2 is backward compatible with the C-Blosc1 API and in-memory format. However, C-Blosc2 buffers are NOT forward-compatible with C-Blosc1. To ensure full API compatibility with the older C-Blosc1 API, define the BLOSC1_COMPAT symbol during compilation.

  9. Understand Blosc2 library and format compatibility

    main

    Blosc2 follows specific compatibility rules for its library versions and the data formats they produce:

    Minor Version Compatibility

    Compatibility between minor versions (e.g., 2.0 and 2.1) is guaranteed to be both backward and forward compatible. Any version of the library can read data produced by another version that differs only by patch or minor version.

    Major Version Compatibility

    • Backward Compatibility: There is an attempt to maintain backward compatibility between major versions, but it is not an absolute guarantee. For example, Blosc2 2.1 is designed to be able to read data produced by Blosc 1.x.
    • Forward Compatibility: There is no guarantee of forward compatibility between major versions. Older major versions (e.g., Blosc 1.x) cannot read data produced by newer major versions (e.g., Blosc 2.x).

  10. Understand the Blosc/Blosc2 Chunk Formats

    main

    Blosc/Blosc2 data is organized into chunks. There are two primary types of chunks:

    1. Regular Chunks: Composed of a header and a blocks section. `+---------+--------+ | header | blocks | +---------+--------+

    `

    1. Lazy Chunks: Used typically when reading from persistent media. They contain meta-information but not the actual compressed data, allowing for selective reading. They consist of a header, bstarts section, and an additional trailer. `+---------+---------+---------+ | header | bstarts | trailer | +---------+---------+---------+

    `

    Note: All integer types are stored in little endian.

  11. Add metadata using meta-layers

    main

    C-Blosc2 allows attaching metadata to data via layers:

    • Regular meta-layers: Stored in the header; intended for small, fixed-length metadata (e.g., NumPy metadata or geospatial info).
    • Variable length meta-layers: Stored in the trailer of a frame; intended for larger metadata (up to 2 GB).

  12. Manage data persistence with frames and super-chunks

    main

    C-Blosc2 uses a hierarchical container model for 64-bit data:

    • super-chunk (schunk): The primary first-class container, composed of smaller chunks (essentially C-Blosc1 32-bit containers).
    • Contiguous frames: A container that backs a super-chunk, allowing chunks to be serialized and stored contiguously in-memory or on-disk. This enables persistence of super-chunks.
    • Sparse frames: An alternative where chunks and metadata are stored separately, allowing for more efficient updates by avoiding 'holes' in monolithic files.