repository·master·Indexed 12 days ago
https://github.com/google/brotliA generic-purpose lossless compression algorithm using LZ77, Huffman coding, and 2nd order context modeling. This repository provides the C implementation, a Python module, and a command-line interface for compression and decompression. It includes tools for analyzing backward reference distance distributions and managing Brotli dictionaries.
The scripts/dictionary directory contains a set of tools designed to manage Brotli dictionaries. These tools allow developers to:
DictionaryData class.research/ directory contains tools designed to analyze backward reference distance distributions in LZ77 compression. These tools help visualize how distances are distributed in large-window compression, which can inform more efficient encoding strategies like delta coding. The tools work by generating *.dist files containing reference data and then visualizing that data as grayscale PGM images (histograms).The *.dist file format stores backward references using a flag-prefixed structure. Each entry consists of either a match length or a position-distance pair. All numbers are 32-bit integers.
Format Structure:
[0| match length]: Flag byte 0 followed by a 4-byte match length.[1| position| distance]: Flag byte 1 followed by a 4-byte position and a 4-byte distance.Standalone copy lengths (flag 0) are allowed but ignored during processing. Position-distance pairs (flag 1) are the primary data points.
To optimize the balance between compression density and speed, you can tune the quality and window size:
-q or --quality): Accepts values from 0 to 11. Higher values result in denser compression but require more time. The -Z or --best flag is a shortcut for -q 11.-w or --lgwin): Sets the LZ77 window size using a logarithmic scale. Valid values are 0 or 10-24 (default is 24). The actual window size is calculated as (2^NUM - 16). A value of 0 allows the compressor to decide the optimal value. Larger windows improve density but increase the memory required by the decoder.# High quality compression with specific window size
brotli -q 11 -w 22 input.txtYou can install the latest release of the Python brotli module from PyPI using pip.
Requirements:
$ pip install brotliThe brotli command-line tool provides three primary operation modes:
.br suffix is appended to the filename.--decompress or -d option. It removes the suffix from the source filename to create the target file.--test or -t option. This performs decompression to verify file integrity but discards the output instead of writing to stdout.Helper Commands:
unbrotli: A shortcut for brotli --decompress.brcat: A shortcut for brotli --decompress --concatenated --stdout (useful for concatenating streams).Input/Output Behavior:
-, brotli reads from standard input.--stdout or --output is used, output filenames are derived from source filenames using a suffix (default .br).--suffix=SUF to change the default suffix.# Compression
brotli input.txt
# Decompression
brotli -d input.txt.br
unbrotli input.txt.br
# Test integrity
brotli -t input.txt.br
# Decompress to stdout
brotli -d -c input.txt.brBrotli provides a Python module that can be installed via pip. You can install the latest stable release or the latest development version directly from the GitHub repository.
# Install the latest release
pip install brotli
# Install the tip-of-the-tree version from GitHub
pip install --upgrade git+https://github.com/google/brotliYou can build and install Brotli using CMake by creating a build directory and configuring the installation prefix. This is useful for integrating Brotli into C++ projects.
$ mkdir out && cd out
$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=./installed ..
$ cmake --build . --config Release --target installThe project provides a Makefile to manage common development tasks. You can build the module in-place, run tests, or clean build artifacts. To make the module available while allowing for live edits to the source files, use make develop to install in setuptools development mode.
$ make # Build the module in-place
$ make test # Test the module
$ make clean # Remove all temporary files and build output
$ make develop # Install the module in "development mode"The project follows the Google Python Style Guide. To automatically format the code, install yapf and then use the make fix command.
$ pip install yapf
$ make fix # Automatically format codeTo install directly from the source directory, use the provided Makefile.
If you want to use a native Brotli installation already present on your system instead of the vendored sources, set the USE_SYSTEM_BROTLI=1 environment variable when building the wheel. This requires the pkg-config utility and the presence of the brotlicommon, brotlienc, and brotlidec components. On Fedora, ensure you have the development package installed.
# Standard source installation
$ make install
# Install using system Brotli instead of vendored sources
$ USE_SYSTEM_BROTLI=1 pip install brotli --no-binary brotli
# Fedora dependency installation
$ dnf install brotli brotli-develFor most Linux and macOS environments, you can install the Brotli CLI and tools using standard package managers:
apt.Homebrew.Note that Brotli is a "stream" format and does not contain meta-information like checksums or uncompressed data length. This means the decoder will not detect if raw ranges of the compressed stream have been modified.
# Debian-based distributions
apt install brotli
# macOS
brew install brotli