DOCSEARCHbyAlgolia
Submit Docs

XZ Utils Documentation

repository·master·Indexed 23 days ago

https://github.com/tukaani-project/xz

High-performance data compression tools and the liblzma library using the .xz format. Features include LZMA2 compression, filter chaining, and multithreaded compression. Documentation covers the liblzma API, lzma_index management for random-access reading, version numbering, and specific configuration for DOS environments including Long Filename (LFN) and Short Filename (SFN) modes.

Tokens
4K
Snippets
7
Records
19
Agent score
78%

What's inside XZ Utils

  1. Understand the XZ Utils Windows package contents

    master

    The Windows package provides command-line tools and the liblzma compression library. It is organized into architecture-specific directories:

    • bin_i686-sse2: 32-bit x86 (requires SSE2 support).
    • bin_x86-64: 64-bit x86-64.

    Executables (*.exe)

    Command-line tools like xz.exe are statically linked against liblzma, meaning they are self-contained. You can copy xz.exe to any directory in your PATH without needing to copy other files from the package.

    Note on decompression tools:

    • xzdec.exe and lzmadec.exe are optimized for small size and are single-threaded. They are slower than xz.exe. Use xz.exe for better performance unless binary size is a critical constraint.

    Libraries and Headers

    • liblzma.dll: The shared version of the compression library.
    • include/: C header files for liblzma, compatible with most C/C++ compilers.
    • doc/examples/: Example programs for basic liblzma usage.

    Requirements

    • Universal CRT (UCRT): All executables and libraries require UCRT. This is included in Windows 10 and later, but can be installed on Windows XP and later.
    • SSE2 Support: Binaries do not work on 32-bit processors that lack SSE2 support (as run-time processor detection is not included).

  2. Understand Short Filename (SFN) mode behavior on DOS

    master

    When LFN support is unavailable or disabled via set lfn=n, xz operates in Short Filename (SFN) mode. This mode changes how file extensions are handled during compression to accommodate DOS filename limitations.

    Compressing to .xz format in SFN mode

    • No extension: Files without an extension get .xz appended.
    • .tar files: *.tar files become *.txz (shorthand for *.tar.xz). This is recognized by xz on all supported operating systems.
    • 1-3 character extensions: The extension is modified so the last character is a dash (-). If the extension is already 3 characters, the last character is lost. Note: These files (*.?- or *.??-) are recognized by xz in LFN mode but not on other operating systems.

    Examples:

    CommandResulting FileDecompression Result
    xz foofoo.xzfoo
    xz foo.tarfoo.txzfoo.tar
    xz foo.cfoo.c-foo.c
    xz read.meread.me-read.me
    xz foo.txtfoo.tx-foo.tx (Note: 3rd char lost)

    Compressing to .lzma format in SFN mode

    • .tar files: *.tar files become *.tlz (shorthand for *.tar.lzma). Recognized by xz on all supported operating systems.
    • Other files: Become *.lzm. The original extension is lost. These are recognized in LFN mode but not on other operating systems.

    Examples:

    CommandResulting FileDecompression Result
    xz -F lzma foofoo.lzmfoo
    xz -F lzma foo.tarfoo.tlzfoo.tar
    xz -F lzma foo.cfoo.lzmfoo (Extension lost)
    xz -F lzma read.meread.lzmread (Extension lost)
    xz -F lzma foo.txtfoo.lzmfoo (Extension lost)

  3. Configure Long Filename (LFN) support on DOS

    master

    On DOS, xz automatically detects if Long Filename (LFN) support is available and uses it by default. If you need to disable LFN support and force Short Filename (SFN) mode, set the lfn environment variable to n.

    LFN Mode Behavior: When LFN is enabled, xz behaves like it does on other operating systems:

    • xz foo.tar -> foo.tar.xz
    • xz -d foo.tar.xz -> foo.tar
    • xz -F lzma foo.tar -> foo.tar.lzma
    set lfn=n

  4. Create an import library for liblzma for MSVC / Visual Studio

    master

    If you want to link your application against liblzma.dll using Microsoft Visual C++ (MSVC), you must first create an import library using the lib command and the liblzma.def file found in the doc directory.

    For 32-bit x86

    lib /def:liblzma.def /out:liblzma.lib /machine:ix86

    For x86-64

    lib /def:liblzma.def /out:liblzma.lib /machine:x64

    Important: If your application does not use the Universal CRT (UCRT), refer to liblzma-crt-mixing.txt for specific guidance.

    lib /def:liblzma.def /out:liblzma.lib /machine:x64

  5. Overview of XZ Utils

    master

    XZ Utils is a general-purpose data-compression library and a set of command-line tools. It natively uses the .xz format, which supports multiple compression algorithms called "filters" (the primary filter being LZMA2).

    Key characteristics:

    • Compression Ratio: Typically produces files ~30% smaller than gzip.
    • Performance: LZMA2 offers high compression ratios at the cost of CPU and RAM, but fast modes compete with bzip2 in speed and RAM usage.
    • Decompression: LZMA2 is reasonably fast to decompress (slower than gzip, but much faster than bzip2), making it ideal for software distribution.
    • Filter Chaining: You can chain up to four filters to improve compression. For example, placing a BCJ (Branch/Call/Jump) filter before LZMA2 can improve the compression of executable files.
    • Multithreading: Supports multithreaded compression (multithreaded decompression is planned for future versions).

  6. How to use liblzma API

    master

    The liblzma API is designed to be similar to the popular zlib library to ease integration into existing applications.

    To get started with liblzma:

    • Reference: The API headers include Doxygen tags for each function and data type.
    • Examples: Tutorial programs are located in the doc/examples directory of the source package or the examples directory in binary packages.
    • Mental Model: If you are familiar with zlib, the liblzma patterns will feel very similar.

  7. Manage .xz Index information with lzma_index

    master

    The lzma_index is an opaque data type used to hold .xz Index information, including Stream Flags and Stream Padding. It can represent a single index or multiple concatenated indexes (via lzma_index_cat).

    Thread Safety:

    • Only one thread may modify an lzma_index at a time (functions taking a non-const pointer).
    • Multiple threads can read from the same lzma_index simultaneously using functions that take a const pointer or by using lzma_index_iter.
    • Each thread must use its own lzma_index_iter instance.

    Lifecycle:

    • Initialize with lzma_index_init().
    • Deallocate with lzma_index_end().
    • Duplicate with lzma_index_dup().

  8. Understand XZ Utils version numbering

    master

    XZ Utils uses the version format X.Y.ZS:

    • X (Major): Incremented when the library API and ABI break.
    • Y (Minor): Incremented when new features are added without breaking API/ABI.
      • An even Y indicates a stable release.
      • An odd Y indicates an unstable (alpha or beta) version.
    • Z (Revision):
      • For stable releases (even Y): Incremented for bug fixes without new features.
      • For unstable releases (odd Y): A simple counter; API/ABI may break.
    • S (Stability): Only present in unstable releases (odd Y) to indicate alpha or beta status.

  9. Test translations for XZ Utils

    master

    To test translations, you can install XZ into a temporary directory using Autotools.

    1. Generate Autotools files (if building from Git):
      ./autogen.sh
    2. Configure and build in a temporary directory:
      mkdir tmp-build
cd tmp-build
../configure --disable-shared --enable-debug --prefix=$PWD/inst
    3. Update and install translations:
      make -C po update-po
make -j"$(nproc)" install
    4. Verify with the debug script:
      bash ../debug/translation.bash | less
# For --list outputs:
bash ../debug/translation.bash | less -S
    5. Test a specific language by setting the LANGUAGE environment variable (use the PO file name without the .po suffix):
      export LANGUAGE=fi
    ./autogen.sh

mkdir tmp-build
cd tmp-build
../configure --disable-shared --enable-debug --prefix=$PWD/inst

make -C po update-po
make -j"$(nproc)" install
bash ../debug/translation.bash | less

export LANGUAGE=fi

  10. Iterate through Blocks and Streams in an lzma_index

    master

    Use lzma_index_iter to traverse the contents of an lzma_index. This allows you to access specific stream and block information.

    Iterator Workflow:

    1. Initialize the iterator with lzma_index_iter_init(iter, index).
    2. Use lzma_index_iter_next(iter, mode) to move to the next element.
    3. Access data via the iter->stream or iter->block structures.
    4. Use lzma_index_iter_rewind(iter) to restart from the beginning.

    Iteration Modes (lzma_index_iter_mode):

    • LZMA_INDEX_ITER_ANY: Get the next Block or Stream.
    • LZMA_INDEX_ITER_STREAM: Get the next Stream (skips remaining blocks in the current stream).
    • LZMA_INDEX_ITER_BLOCK: Get the next Block. If the current stream is exhausted, it moves to the first block of the next stream.
    • LZMA_INDEX_ITER_NONEMPTY_BLOCK: Like LZMA_INDEX_ITER_BLOCK, but skips blocks with an uncompressed_size of zero.

    Random Access: Use lzma_index_iter_locate(iter, target) to jump the iterator to the specific uncompressed offset target within a stream.

  11. Use debug tools for XZ Utils development

    master

    The debug/ directory contains small utility programs intended to assist in debugging XZ Utils.

    Important usage notes:

    • Not for installation: These tools are not intended to be installed as part of the system.
    • Requires modification: They are often intended to be edited directly in the source code to suit specific debugging needs.
    • Not for production or examples: These programs take shortcuts and ignore many errors (specifically I/O errors). They should not be used as reference implementations for correct programming practices, and bugs related to their error-handling shortcuts should not be reported.

  12. Troubleshoot DOS-specific xz issues

    master

    Safety Checks

    xz includes safety mechanisms to prevent data loss or system issues on DOS:

    • Self-Overwrite Protection: If the generated output filename would overwrite the input file (common with overlong filenames in SFN mode), xz will refuse to run, even if --force is used. Example: xz -S x foo.texinfo might attempt to write to foo.tex, which is the same file in SFN mode.
    • Special File Protection: xz refuses to operate if the output filename is a DOS special file like con or prn, even with --force.

    Known Limitations

    • Dosbox: xz does not necessarily work in Dosbox. It is expected to work in DOSEMU.
    • Signal Handling (Windows XP): On Windows XP Command Prompt, pressing Ctrl-c or Ctrl-Break may fail to remove incomplete target files due to SIGINT handling issues. This is not an issue on Windows 95/98/ME.