DOCSEARCHbyAlgolia
Submit Docs

imagecodecs

repository·master·Indexed 41 days ago

https://github.com/cgohlke/imagecodecs

A collection of image and data codecs including Bitshuffle for high-performance bit-level rearrangement (often used with HDF5, LZ4, and ZSTD), bcdec for Block Compression (BC1-BC7) image decompression, and LZF for high-speed compression.

Tokens
1.7K
Snippets
4
Records
12
Agent score
52%

What's inside imagecodecs

  1. Overview of the bcdec library

    master
    bcdec is a small, header-only C library designed to decompress blocks of BC compressed images. It is inspired by the stb libraries and is characterized by having zero external dependencies and no use of runtime functions. The library focuses on a minimal footprint, avoiding memory allocation and minimizing stack usage, rather than prioritizing raw execution speed.

  2. Optimize Bitshuffle Performance

    master

    To get the best results from Bitshuffle, follow these data arrangement guidelines:

    • Correlation: Ensure adjacent elements over the fastest varying index of your dataset are highly correlated (similar values). For multi-dimensional datasets, order data so the fastest varying dimension is the one with the highest correlation.
    • Data Type Size: For highest throughput, use data types that are 64 bytes or smaller. If using large compound types, consider adding a dimension to the dataset instead.
    • Instruction Set Alignment:
      • To utilize SSE2, use a data type size that is a multiple of 2 bytes.
      • To utilize AVX2, use a data type size that is a multiple of 4 bytes.

  3. Decompression output specifications for bcdec

    master

    The output format of the decompression depends on the input BC format:

    • BC1, BC2, BC3, BC7: Decompress into 4x4 RGBA blocks (8-bit per component, 32-bit pixel).
    • BC4, BC5: Decompress into 4x4 R/RG blocks (8-bit or 16-bit per component).
    • BC6H: Decompress into 4x4 RGB blocks using either 32-bit float or 16-bit "half" per component (96-bit or 48-bit pixel).

  4. Install Bitshuffle for Python

    master

    You can install Bitshuffle using pip.

    Note on Binary Wheels: On Linux and macOS x86_64, binary wheels are available and built with AVX2 support. These require processors from 2015 onwards (Intel Haswell, AMD Excavator, or later). If you are using an older processor, the build will crash with SIGILL. To avoid this, force a source build using the --no-binary=bitshuffle flag.

    Source Installation Requirements:

    • Python 2.7+ or 3.3+
    • HDF5 1.8.4 or later
    • h5py (HDF5 for Python)
    • numpy
    • cython

    To enable ZSTD support during a source installation, you must first update the git submodules:

    pip install bitshuffle

# To force source build for older CPUs:
pip install bitshuffle --no-binary=bitshuffle

# To enable ZSTD support before installation:
git submodule update --init
python setup.py install [--h5plugin [--h5plugin-dir=spam] --zstd]

  5. Overview of the LZF compression algorithm

    master
    LZF is an extremely fast compression algorithm designed for applications that prioritize speed over high compression ratios. It is nearly as fast as a memcpy and is particularly effective for repetitive data. The library is self-contained, very small, and written in ISO-C with no external dependencies.

  6. Configure HDF5 Plugin Path

    master
    If you are using the dynamically loaded HDF5 filter (which allows Bitshuffle to be used outside of Python in tools like h5dump), you must set the HDF5_PLUGIN_PATH environment variable to the directory containing the plugin (the value used in --h5plugin-dir during installation) or use the HDF5 default search location: /usr/local/hdf5/lib/plugin.

  7. Integrate LZF into C projects

    master
    Because the library is written in ISO-C and has no external dependencies, you can easily integrate it into your C projects by simply #include-ing the header file. No changes to your Makefile or complex library builds are required. For specific function signatures and usage details, refer to the lzf.h file.

  9. Use Bitshuffle in Rust via HDF5 Plugin

    master

    To open HDF5 files compressed with Bitshuffle in Rust, use the hdf5-bitshuffle crate.

    1. Add to Cargo.toml:
    [dependencies]

hdf5-bitshuffle = "0.9"
    1. Register the plugin at the start of your application.
    use hdf5_bitshuffle::register_bitshuffle_plugin;

fn main() {
    register_bitshuffle_plugin();
}

  10. Use Bitshuffle in Java

    master

    Bitshuffle routines are ported to snappy-java.

    1. Add the dependency to your pom.xml:
    <dependency>
  <groupId>org.xerial.snappy</groupId>
  <artifactId>snappy-java</artifactId>
  <version>1.1.3-M1</version>
</dependency>
    1. Import and use org.xerial.snappy.BitShuffle in your code.
    import org.xerial.snappy.BitShuffle;

int[] data = new int[] {1, 3, 34, 43, 34};
byte[] shuffledData = BitShuffle.bitShuffle(data);
int[] result = BitShuffle.bitUnShuffleIntArray(shuffledData);

  11. Use Bitshuffle with h5py in Python

    master

    The bitshuffle.h5 module provides convenience functions for creating datasets with Bitshuffle compression. For h5py version 2.5.0 and later, you can use the high-level create_dataset interface.

    Bitshuffle is identified by the HDF5 filter number 32008. To achieve actual compression (rather than just bit-rearrangement), you must pair Bitshuffle with a compression algorithm like LZ4 or ZSTD via compression_opts.

    Compression Options Format:

    • LZ4: (block_size, bitshuffle.h5.H5_COMPRESS_LZ4)
    • ZSTD: (block_size, bitshuffle.h5.H5_COMPRESS_ZSTD, COMP_LVL)

    Note: Setting block_size = 0 lets Bitshuffle choose its own value.

    import h5py
import numpy
import bitshuffle.h5

filename = "example.h5"
f = h5py.File(filename, "w")

# block_size = 0 lets Bitshuffle choose its value
block_size = 0

dataset = f.create_dataset(
    "data",
    (100, 100, 100),
    compression=bitshuffle.h5.H5FILTER,
    compression_opts=(block_size, bitshuffle.h5.H5_COMPRESS_LZ4),
    dtype='float32',
)

# create some random data
array = numpy.random.rand(100, 100, 100).astype('float32')
dataset[:] = array

f.close()