PyroWave Video Codec

PyroWave is an intra-only video codec (effectively a still-image codec) optimized for extremely fast GPU encoding and decoding using Vulkan compute shaders. It is designed for high-bandwidth, low-latency use cases like local network game streaming (targeting bit-rates of 200+ mbit/s).

Key Technical Features:

• Colorspace: Supports YCbCr 4:2:0 and 4:4:4.
• Wavelets: Uses Discrete Wavelet Transform with CDF 9/7 filter (similar to JPEG2000).
• Rate Control: Supports exact maximum bitrate targeting per encoded image.
• Entropy Coding: Uses a simplified, highly parallelizable entropy coding method to maximize throughput.
• Robustness: Encodes 64x64 blocks of coefficients in isolation, providing high resilience against packet loss.

This is an AI-assisted port of Pyrowave specifically for Apple platforms using native Metal. Unlike the standard implementation, this version does not use Granite code; instead, it uses native Metal and converts shaders to MSL (Metal Shading Language).

Key characteristics include:

• Simplified API: The API has been stripped of Vulkan-specific patterns, utilizing native Metal objects for GPU decoding instead.
• Performance: On Apple hardware, this implementation uses FP32 math with FP16 storage in the shaders to achieve optimal speed and accuracy. It is approximately twice as fast as the Vulkan implementation running on KosmicKrisp on macOS.
• License: Covered by the standard MIT license.

Decoding a bitstream follows a four-stage pipeline:

1. Decoding wavelet coefficients: Extracting integer coefficients from the bitstream.
2. Scaling wavelet coefficients (De-quantization): Converting integer coefficients to floating-point values using scale factors and a deadzone.
3. Inverse DWT: Performing the Inverse Discrete Wavelet Transform per component (Y, Cb, Cr).
4. DC shift fixup & clamping: Shifting values by +0.5 and clamping the final result to the [0, 1] range.

Note: The first two stages and the last two stages can be fused during implementation for efficiency.

PyroWave uses a 5-level Discrete Wavelet Transform (DWT) with the irreversible CDF 9/7 filter (the same as JPEG2000).

Decomposition Levels

The transform decomposes a single image component (Y, Cb, or Cr) into four sub-bands (LL, HL, LH, HH) and then subsamples. This repeats until the final LL4 sub-band is reached. The full-resolution image is considered LL-1.

Implementation Details

• Lifting Scheme: The CDF 9/7 filter can be implemented using a lifting scheme.
• Edge Handling: Input pixels are mirrored on the edges for both forward and inverse transforms. On GPUs, this can be implemented using VK_SAMPLER_ADDRESS_MODE_MIRRORED_REPEAT and textureGather().
• Precision Requirements: Intermediate floating-point values can exceed +/- 1.0. Values above +/- 4.0 may be saturated. Inf and NaN are not permitted.

PyroWave is a byte-oriented image codec designed for high-throughput, low-latency transmission over IP networks. It organizes data into a series of packets.

Key characteristics:

• Intra-only video: It is fundamentally a still-image codec used for video as a sequence of independent images (no inter-prediction).
• Endianness: Multi-byte values use little-endian layout.
• Implementation target: Designed to be implemented efficiently using (Vulkan) compute shaders on consumer GPUs.
• Precision: The decoding process is not bit-exact due to the use of the floating-point CDF 9/7 wavelet filter. The inverse transform requires at least FP16 precision.

To build PyroWave with its standalone C API, you must first check out a portion of the Granite dependency using the provided script. The API is currently under development and the ABI is not yet stable.

1. Checkout Granite dependency:

   checkout_granite.sh

2. Build using CMake and Ninja:

   mkdir build
   cd build
   cmake .. -DCMAKE_INSTALL_PREFIX=output -DCMAKE_BUILD_TYPE=Release -G Ninja
   ninja install

Supported platforms include Linux, MinGW, msys2, and MSVC. For Steam Runtime, use build-steamrt.sh.

# Checkout dependency
checkout_granite.sh

# Build and install
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=output -DCMAKE_BUILD_TYPE=Release -G Ninja
ninja install

To avoid running expensive bitrate sweeps in real-time, you can use the polynomial coefficients generated by the regress.py script. These coefficients allow you to solve for the required bitrate given:

• Resolution
• Viewing distance
• Chroma 4:4:4 status
• Target quality

The regression uses 7th order polynomials to estimate curves with approximately 1% accuracy. The results are provided as a C header file, pyrowave_regression_results.h, which can be directly included in your application to perform these lookups efficiently.

For local development including sample and test applications, you should clone the full Granite repository before running CMake. Use the -DPYROWAVE_DEVEL=ON flag to enable the full build.

git clone --depth 1 --recursive --shallow-submodules https://github.com/Themaister/Granite Granite

git clone --depth 1 --recursive --shallow-submodules https://github.com/Themaister/Granite Granite

# When running CMake, use:
# cmake .. -DPYROWAVE_DEVEL=ON

To build PyroWave for Android, you need an installed NDK. You must specify the NDK toolchain file and the target ABI (e.g., arm64-v8a).

Note: The example below uses NDK_VERSION=29.0.14206865. You should adjust this to match your installed NDK version.

# Replace NDK_VERSION as needed
NDK_VERSION=29.0.14206865

cmake .. -DCMAKE_TOOLCHAIN_FILE=$ANDROID_HOME/ndk/$NDK_VERSION/build/cmake/android.toolchain.cmake \
    -DCMAKE_BUILD_TYPE=Release \
    -DANDROID_ABI=arm64-v8a \
    -G Ninja -DCMAKE_INSTALL_PREFIX=output
ninja install

The pyrowave-evaluator tool implements a double-blind testing scheme (based on BT.500) to evaluate the perceived quality of PyroWave streams.

Testing Workflow:

1. The subject is shown a sequence: 10s lossless reference $\rightarrow$ 3s mid-gray $\rightarrow$ 10s test clip $\rightarrow$ 3s mid-gray $\rightarrow$ 10s lossless reference $\rightarrow$ 3s mid-gray $\rightarrow$ 10s test clip $\rightarrow$ 3s mid-gray.
2. During the final 8-second mid-gray period, the subject votes on quality using a scale of 1 to 5:
   • 5: Imperceptible
   • 4: Perceptible, but not annoying
   • 3: Slightly annoying
   • 2: Annoying
   • 1: Very annoying

Target quality for streaming is typically between 4 and 5.

Note: For accurate results, use a screen native to the test resolution (e.g., Steam Deck for 720p, 4K monitor for 4K content).