DOCSEARCHbyAlgolia
Submit Docs

Zimtohrli Psychoacoustic Perceptual Metric

repository·main·Indexed 42 days ago

https://github.com/google/zimtohrli

A psychoacoustic perceptual metric for audio compression that quantifies human-perceived differences between audio signals. It uses a multi-stage signal processing pipeline—including a 3rd Order Complex Gammatone Filterbank, physiological modeling, Dynamic Time Warping (DTW), and a modified Neurogram Similarity Index Measure (NSIM)—to model the human auditory system. Provided as a C++ library with a Go wrapper (goohrli) and a CLI compare tool.

Tokens
11.4K
Snippets
25
Records
37
Agent score
48%

What's inside Zimtohrli

  1. What is Zimtohrli?

    main

    Zimtohrli is a psychoacoustic perceptual metric designed to quantify the human-observable difference between two audio signals. It focuses on the proximity of just-noticeable-differences (JND) to provide an accurate model of human-subjective similarity. This makes it useful for high-quality audio compression and audio-related machine learning models.

    Key characteristics:

    • Input: Audio signals sampled at 48 kHz (floats between -1 and 1).
    • Output: A scalar distance value between 0 (identical) and 1 (maximally different).
    • Performance: Can compare approximately 70 seconds of audio per second on a single 2.5GHz core.

  2. How the Zimtohrli signal processing pipeline works

    main

    Zimtohrli models the human auditory system through a four-stage pipeline:

    1. 3rd Order Complex Gammatone Filterbank: Processes input through 128 bins of 3rd order filters (center frequencies 24.349 Hz to 19658.3 Hz) using a rotating phasor algorithm. It uses bandwidth-dependent exponential windowing to model the basilar membrane.
    2. Physiological Modeling: Applies transformations including a 32-point FIR resonator model (ear drum/middle ear), a 32-point FIR filter for direct absorption (e.g., skull), a logarithmic loudness transformation with frequency-dependent gains, and a frequency-dependent bias energy to model hearing thresholds.
    3. Temporal Alignment: Uses Dynamic Time Warping (DTW) with a perceptually-motivated cost function (combining spectral distance and temporal distortion) to handle misalignments between reference and test signals.
    4. Perceptual Similarity Computation: Compares aligned spectrograms using a modified Neurogram Similarity Index Measure (NSIM). NSIM computes windowed statistics (mean, variance, covariance) over 8 temporal frames and 5 frequency channels.

  3. How correlation scoring works

    main

    The correlation testing framework evaluates metrics against human-subjective datasets using two distinct methods:

    • Mean Opinion Score (MOS) datasets: The system calculates the Spearman rank correlation coefficient between the predicted scores and human ratings. A value closer to 1.0 indicates better alignment with human perception.
    • Just Noticeable Difference (JND) datasets: The system determines classification accuracy. It finds an optimal threshold to maximize correct predictions of whether a difference between two signals is audible or not. The score is the percentage of correct classifications.

  4. Run correlation tests for metric validation

    main

    Zimtohrli includes a framework to validate how well the metric correlates with human perception (MOS or JND ratings).

    Workflow

    1. Install external metrics (optional):

      ./install_external_metrics.sh /path/to/destination

    2. Acquire datasets using the tools in go/bin/ (e.g., coresvnet, perceptual_audio, sebass_db, odaq, tcd_voip).

    3. Calculate metrics: Use the score.go tool to calculate Zimtohrli and other metrics (like VISQOL) against your datasets.

      go run go/bin/score/score.go -calculate "/path/to/datasets/*" -calculate_zimtohrli -calculate_visqol

    4. Generate correlation report:

      go run go/bin/score/score.go -report "/path/to/datasets/*" > correlation_report.md

    Scoring Logic

    • MOS datasets: Uses Spearman rank correlation coefficient (higher is better, closer to 1.0).
    • JND datasets: Uses classification accuracy based on an optimal threshold for audible differences.

  5. Install dependencies and build Zimtohrli

    main

    Zimtohrli requires C++17 and is designed for Debian-like environments.

    1. Install System Dependencies

    Run the following to install the necessary build tools and libraries for the core, compare tool, and wrappers (Python/Go):

    sudo apt install -y cmake ninja-build clang clang-tidy libogg-dev libvorbis-dev libflac-dev libopus-dev libasound2-dev libglfw3-dev libsoxr-dev golang-go python3 xxd zlib1g-dev ffmpeg

    2. Configure and Build

    Standard Build:

    ./configure.sh
(cd build && ninja)

    Address Sanitizer (ASan) Build:

    ./configure.sh asan
(cd asan_build && ninja)

    Debug Build:

    ./configure.sh debug
(cd debug_build && ninja)

    3. Run Tests

    (cd build && ninja && ninja test)

  6. Install the Zimtohrli Go wrapper (goohrli)

    main

    The goohrli package is a Go wrapper for the Zimtohrli C++ library using cgo.

    Prerequisites

    On Debian-like systems, you must install the following C++ and audio development dependencies before installing the Go package:

    sudo apt install -y libc++-dev libc++abi-dev libflac-dev libogg-dev libvorbis-dev libopus-dev

    Installation

    To add the wrapper to your Go module, use go get:

    go get github.com/google/zimtohrli/go/goohrli

  7. Use the compare command line tool

    main

    The compare tool is a CLI utility used to compare two WAV files using the Zimtohrli metric.

    Installation

    Install the binary using go install:

    go install github.com/google/zimtohrli/go/bin/compare

    Usage

    Run the tool by providing the paths to your reference WAV file and your distorted WAV file. Ensure the binary is in your $GOPATH/bin:

    $GOPATH/bin/compare -path_a reference.wav -path_b distortion.wav

  8. Import pyohrli from the build directory

    main

    If you are running scripts from within the repository structure (e.g., in a Jupyter notebook), you may need to manually append the build directory to your sys.path to locate the pyohrli Python bindings.

    import sys
import os
sys.path.append(os.path.join("..", "..", "build"))
import pyohrli

  9. Prepare data for MOS mapping optimization

    main

    Before performing Mean Opinion Score (MOS) mapping optimization, you must generate and export your scores using the go/bin/score tool.

    1. Generate scores for your target datasets using the go/bin/score tool.
    2. Export these scores to a JSON file located at /tmp/scores.json using the same tool.

    The expected JSON structure involves a list of datasets, where each dataset contains a References key holding a list of reference objects. Each reference object must contain a Distortions list, and each distortion must include both Zimtohrli and MOS keys within its Scores object.

    # Example conceptual workflow
go/bin/score --output /tmp/scores.json

  10. Use the tcd_voip CLI to create a study from the TCD VOIP dataset

    main

    The tcd_voip tool automates the creation of a Zimtohrli study using the TCD VOIP dataset. It processes audio files and subjective MOS (Mean Opinion Score) data provided in a specific CSV format.

    Prerequisites

    1. Download the TCD VOIP dataset zip and unpack it.
    2. Open TCD VOIP - Test Set Conditions and MOS Results.xlsx in a spreadsheet application.
    3. Export the Subjective Test Scores tab as a .csv file.
    4. Place the exported .csv file in the same directory as the unpacked dataset files.

    Usage

    Run the tool by specifying the source directory (containing the dataset and CSV) and a destination directory for the processed study files.

    tcd_voip -source /path/to/unpacked_dataset -dest /path/to/destination_dir

  11. Use the odaq CLI for audio quality troubleshooting

    main

    The odaq CLI tool is used to download, process, and prepare the ODAQ listening test datasets for use with Zimtohrli. It automates the downloading of MUSHRA test files from Zenodo, converts audio files (re-coding them), and populates a Zimtohrli study directory with the resulting references and distortion scores.

    To use the tool, you must specify a destination directory where the processed study data will be stored. You can also control the parallelism of the audio conversion process.

    # Basic usage with default worker count
./odaq -dest ./my_study_data

# Usage with a specific number of parallel workers
./odaq -dest ./my_study_data -workers 4

  12. Use the `score` CLI to manage audio metric datasets

    main

    The score CLI tool is used to manage listening test datasets. It allows you to calculate psychoacoustic metrics (like Zimtohrli or ViSQOL), correlate scores across different metrics, generate leaderboards, and produce reports from audio databases. Most commands accept a glob pattern to specify directories containing databases.

    Core Tasks

    • Calculate Metrics: Use -calculate with metric-specific flags to compute scores for audio files.
    • Correlate Scores: Use -correlate to compute correlation tables between different score types in a dataset.
    • Generate Reports & Leaderboards: Use -report or -leaderboard to summarize results across multiple bundles.
    • Analyze Accuracy: Use -accuracy to provide Just Noticeable Difference (JND) accuracy for datasets.
    • Inspect Data: Use -details to view the raw JSON details of bundles.
    # Example: Calculate Zimtohrli scores for databases in ./data
./score -calculate "./data/*" -calculate_zimtohrli

# Example: Correlate scores in a directory
./score -correlate "./data/results/*"

# Example: Generate a leaderboard for datasets
./score -leaderboard "./data/bundles/*"