rapidgzip

rapidgzip is a tool for parallelized decompression of gzip files, supporting fast random access.

Key features include:

• Universal Compatibility: Unlike tools like bgzip which only work with files they produced, rapidgzip works with almost any gzip file, including those produced by standard GNU gzip.
• Parallel Decompression: It can utilize all available CPU cores to speed up decompression.
• Random Access: Through the Python RapidgzipFile class, you can seek inside gzip files without decompressing the entire file first.
• Performance Optimization: Uses a least-recently-used (LRU) cache and a parallelized prefetcher to speed up seeking, similar to indexed_gzip but with higher memory usage for better speed.

rapidgzip performance varies depending on whether an index is present:

• With Index (--import-index): Generally faster. It can delegate decompression to ISA-l or zlib. Parallelization is more effective because the serializing window propagation step is not required.
• Without Index: Slower. rapidgzip must use its own custom-written gzip decompression engine and perform a serializing window propagation step.

The easiest way to install rapidgzip is using pip. It is recommended to upgrade pip first to ensure compatibility with newer manylinux wheels.

python3 -m pip install --upgrade pip
python3 -m pip install rapidgzip

rapidgzip is the default backend for ratarmount (since version 0.14.0). You can use ratarmount to mount single gzip files as if they were regular filesystems, enabling parallel decoding and fast random access via the mount point.

# Install ratarmount
python3 -m pip install --user ratarmount

# Mount a gzip file
ratarmount sample.gz mounted

# Access the file via the mount point
cat mounted/sample

# Perform random seeking using standard tools like dd
dd if=mounted/sample bs=$(( 1024 * 1024 )) \
       iflag=skip_bytes,count_bytes skip=$(( 2 * 1024 * 1024 * 1024 )) count=$(( 1024 * 1024 )) | wc -c

You can use rapidgzip in Python to perform parallelized decompression with fast random access. The library provides a file-like interface that supports seek(), read(), and close().

Note: The first call to seek() may take some time because it must ensure the block offset list is complete by decoding the gzip file.

from rapidgzip import RapidgzipFile
import os

file = RapidgzipFile("example.gz", parallelization=os.cpu_count())

# Use it like a normal file
file.seek(123)
data = file.read(100)
file.close()

The rapidgzip command-line tool is used for parallelized decompression of gzip files. By default, it can decompress from standard input to standard output or to a specified output file. To achieve maximum performance, use the -P (or --decoder-parallelism) flag to enable parallel decoding. Passing 0 to this flag allows the tool to automatically determine the optimal number of threads based on your system.

# Decompress a file
rapidgzip -d file.gz

# Decompress a file in parallel (automatic thread count)
rapidgzip -d -P 0 file.gz

If you need to build from source on Debian-based systems, you must first install the necessary build dependencies:

sudo apt install git gcc g++ python3 python3-dev python3-pip python3-build python3-venv nasm

To install the latest unreleased development version directly from GitHub:

python3 -m pip install --force-reinstall 'git+https://github.com/mxmlnkn/rapidgzip.git@main#egginfo=rapidgzip&subdirectory=python/rapidgzip'

To build a local wheel manually:

git clone --recursive https://github.com/mxmlnkn/rapidgzip.git
cd rapidgzip/python/rapidgzip/
rm -rf dist
python3 -m build
python3 -m pip install --force-reinstall --user --break-system-packages dist/rapidgzip-*.whl

Creating the list of gzip blocks requires decoding the file completely, which can be slow. To avoid this overhead when reopening a file, you can export the block offset map to a file and import it later.

import os
import rapidgzip

index_path = "example.gz.gzindex"

# Exporting the index
with rapidgzip.open("example.gz", parallelization=os.cpu_count()) as file:
    file.seek(123)
    data = file.read(100)
    file.export_index(index_path)

# Importing the index
with rapidgzip.open("example.gz", parallelization=os.cpu_count()) as file:
    file.import_index(index_path)
    file.seek(123)
    data = file.read(100)

rapidgzip is implemented in C++ and is a header-only library to simplify compilation and support heavy template usage. It supports streaming uncompressed data with a predefined buffer size.

You can integrate it into CMake projects using FetchContent or add_subdirectory.

Use the --analyze flag to inspect the internal structure of a gzip file, including information about block types and the distribution of deflate blocks.

rapidgzip --analyze file.gz

You can use rapidgzip.open() to wrap an existing in-memory file-like object (such as io.BytesIO) for indexed reading.

import io
import os
import rapidgzip as rapidgzip

with open("example.gz", "rb") as file:
    in_memory_file = io.BytesIO(file.read())

with rapidgzip.open(in_memory_file, parallelization=os.cpu_count()) as file:
    file.seek(123)
    data = file.read(100)

The --ranges option allows for partial decompression of a gzip file by specifying exact byte offsets or line-based ranges. This is useful for random access patterns.

Example syntax: 10@0,1KiB@15KiB,5L@20L decompressing:

• The first 10 bytes.
• 1024 bytes starting at offset 15 KiB.
• 5 lines after skipping the first 20 lines.

rapidgzip -d --ranges "10@0,1KiB@15KiB,5L@20L" file.gz