pgzip

pgzip provides parallelized gzip operations:

• Compression: Splits data into blocks that are compressed in parallel. This is highly effective for large datasets (over 1MB).
• Decompression: Decompresses data ahead of the current reader in a separate goroutine. This acts as a buffer, ensuring that reads from your application are non-blocking if the decompressor can keep up. CRC calculation also runs in a separate goroutine.

Because pgzip produces standard gzip files, you can compress with pgzip and decompress with any standard gzip tool, and vice versa.

You can use pgzip as a direct replacement for the standard library compress/gzip by changing your import statement. The API is designed to be compatible.

Replace:

import "compress/gzip"

With:

import gzip "github.com/klauspost/pgzip"

To install pgzip, use the following Go command:

go get github.com/klauspost/pgzip/...```

You may also need to update the `github.com/klauspost/compress` dependency:

```bash
go get -u github.com/klauspost/compress

pgzip decompression is modified to decompress ahead of the current reader, making reads non-blocking as long as the decompressor can keep ahead.

To specify a custom readahead behavior, use pgzip.NewReaderN instead of the standard NewReader.

Parameters:

• r: The source io.Reader.
• blockSize: The size of each block decoded.
• blocks: The maximum number of blocks to be decoded ahead.

This allows you to tune how much data is buffered and processed in the background.

Use NewWriter to create a new Writer with the default compression level, or NewWriterLevel to specify a custom compression level. The Writer implements io.WriteCloser.

Important:

• It is the caller's responsibility to call Close() when finished. Writes may be buffered and not flushed until Close() or Flush() is called.
• If you need to set fields in the Writer.Header (like Name, Comment, or Extra), you must do so before the first call to Write() or Close().
• Header strings (Name and Comment) must be UTF-8 and contain only Latin-1 (ISO 8859-1) characters. Using NUL or non-Latin-1 runes will result in an error during Write().

Use NewReaderN(r io.Reader, blockSize, blocks int) to fine-tune the decompression performance by controlling the prefetching behavior.

• blockSize: The approximate size of each decompressed block.
• blocks: The number of blocks to prefetch.

Default values (if provided values are too small) are blockSize = 250000 and blocks = 16.

Use NewReader(r io.Reader) to create a new Reader that decompresses data from the provided reader. This implementation uses buffering and prefetching to improve performance. It is the caller's responsibility to call Close() on the Reader when finished to release resources.

Note: The implementation may read more data from the underlying reader than requested due to its internal buffering and readahead mechanism.

You can fine-tune the performance of the Writer by controlling the block size and the number of parallel compression tasks.

By default, pgzip uses a block size of 1 MB (defaultBlockSize) and processes up to runtime.GOMAXPROCS(0) blocks in parallel.

Use SetConcurrency(blockSize, blocks) to adjust these values.

• blockSize: The approximate size of each data block. Must be greater than tailSize (16384 bytes).
• blocks: The number of blocks to be processed in parallel. Must be greater than 0.

The Write method writes compressed data to the underlying io.Writer.

Key behaviors:

• Non-blocking: The function returns quickly by handing off data to background goroutines. A nil error does not guarantee that compression has finished or succeeded; it only means the data was successfully queued.
• Buffer Safety: The byte slice p passed to Write is copied. You are free to reuse the buffer immediately after the function returns.
• Error Handling: Errors occurring during background compression are reported asynchronously. To catch these errors, you must call Flush() or Close(), which are guaranteed to return any errors encountered up to that point.