DOCSEARCHbyAlgolia
Submit Docs

GTCRN Speech Enhancement

repository·main·Indexed 20 days ago

https://github.com/xiaobin-rong/gtcrn

An ultra-lightweight speech enhancement model (48.2K parameters) designed for high-performance speech cleaning with low computational overhead. It supports real-time streaming inference via ONNX Runtime and includes a LADSPA plugin (gtcrn-ladspa-ort) for Linux systems, compatible with PipeWire for system-wide noise suppression.

Tokens
3.6K
Snippets
15
Records
19
Agent score
72%

What's inside GTCRN

  1. Overview of GTCRN

    main
    GTCRN (Grouped Temporal Convolutional Recurrent Network) is an ultra-lightweight speech enhancement model designed for minimal computational resource usage. It features approximately 48.2K parameters and 33.0 MMACs per second. It is designed to outperform lightweight models like RNNoise while remaining competitive with much larger baseline models in terms of speech quality metrics (SISNR, PESQ, STOI, etc.).

  3. Use pre-trained GTCRN models

    main

    Pre-trained models for GTCRN are available in the checkpoints directory. These models have been trained on the following datasets:

    • DNS3
    • VCTK-DEMAND

    To run inference using these models, use the infer.py script provided in the repository.

    python infer.py

  4. Build the GTCRN LADSPA Plugin

    main

    You can build the GTCRN LADSPA plugin using three different methods depending on your requirements for portability, size, or development speed:

    1. Minimal Build (Static Link - Docker): Produces the smallest, single-file plugin (~12MB) with no external dependencies. Best for distribution.

      • Run ./build-minimal-docker.sh to build the minimal runtime (takes 5-20 minutes).
      • Run ./build.sh minimal to build the plugin.

    2. Static Build (Download - Bundled): Downloads and bundles the official Microsoft ONNX Runtime. Easier than Docker but results in a larger file (~22MB).

      • Run ./build.sh static.

    3. Dynamic Build (System Lib): Fastest for development, but requires onnxruntime to be already installed on your system.

      • Run ./build.sh dynamic.
    # Minimal Build
./build-minimal-docker.sh
./build.sh minimal

# Static Build
./build.sh static

# Dynamic Build
./build.sh dynamic

  5. Perform streaming inference with GTCRN

    main
    GTCRN supports streaming inference, which is optimized for real-time applications. Implementation details and demonstrations for streaming can be found in the stream folder. On hardware such as a 12th Gen Intel(R) Core(TM) i5-12400 CPU @ 2.50 GHz, the model achieves a real-time factor (RTF) of 0.07.

  6. Configure GTCRN as a PipeWire Noise Suppression Filter

    main

    To use the GTCRN plugin as a system-wide noise suppression filter in PipeWire, follow these steps:

    1. Install: Copy the compiled .so file to your LADSPA directory (e.g., /usr/lib/ladspa/libgtcrn_ladspa.so).
    2. Configure: Create a PipeWire filter-chain configuration file at ~/.config/pipewire/filter-chain.conf.d/gtcrn.conf.
    3. Run: Start PipeWire using the configuration file.

    Plugin Details for Configuration:

    • Plugin Name: libgtcrn_ladspa
    • Label: gtcrn_mono
    • Controls:
      • Strength: A value from 0.0 (Original) to 1.0 (Fully Processed).
      • Model (0=Light 1=Full): Selects the model complexity (0 for Light, 1 for Full).
    context.modules = [
    {
        name = libpipewire-module-filter-chain
        args = {
            node.description = "Noise Canceling Microphone (GTCRN)"
            media.name = "Noise Canceling Microphone (GTCRN)"
            filter.graph = {
                nodes = [
                    {
                        type = ladspa
                        name = "gtcrn"
                        plugin = "libgtcrn_ladspa"
                        label = "gtcrn_mono"
                        control = {
                            # Strength: 0.0 (Original) to 1.0 (Fully Processed)
                            "Strength" = 1.0
                            "Model (0=Light 1=Full)" = 1
                        }
                    }
                ]
            }
            audio.channels = 1
            capture.props = {
                node.passive = true
            }
            playback.props = {
                media.class = "Audio/Source"
            }
        }
    }
]

  7. Use the GTCRN LADSPA plugin for Linux

    main

    A LADSPA plugin is available for filtering live audio on Linux systems using pipewire. This plugin is implemented via ONNX Runtime (package gtcrn-ladspa-ort).

    Note: LADSPA support is currently experimental and is maintained by community contributor Bruno Gonçalves.

  8. Select a GTCRN model type

    main

    The gtcrn-ladspa-ort package provides two model variants via the ModelType enum. You can choose between a faster, lighter model or a full-quality model.

    • ModelType::Simple: A lighter, faster model (gtcrn_simple.onnx).
    • ModelType::Full: A full-quality model (gtcrn.onnx).

    You can also convert a control value (float) to a ModelType using ModelType::from_control(value), where values $\ge 0.5$ select Full and values $< 0.5$ select Simple.

    use gtcrn_ladspa_ort::{ModelType};

let simple_model = ModelType::Simple;
let full_model = ModelType::Full;

// Convert from a control value
let type_from_val = ModelType::from_control(0.7);
assert_eq!(type_from_val, ModelType::Full);

  9. Switch model types at runtime

    main

    You can switch between ModelType::Simple and ModelType::Full on an existing GtcrnModel instance using set_model_type. Note that switching models will reload the ONNX session and automatically reset the internal recurrent state to zeros to ensure consistency.

    let mut model = GtcrnModel::new(ModelType::Simple);

// Switch to full quality model
// This reloads the session and resets the state
model.set_model_type(ModelType::Full);

  10. Retrieve the LADSPA plugin descriptor via get_ladspa_descriptor

    main

    The function get_ladspa_descriptor is the primary entry point for LADSPA hosts to discover the plugin's capabilities, ports, and metadata. It must be called with index == 0 to return the PluginDescriptor.

    Returns: Option<PluginDescriptor> containing the plugin's unique ID, name, maker, copyright, and port definitions.

    #[no_mangle]
pub extern "C" fn get_ladspa_descriptor(index: u64) -> Option<PluginDescriptor> {
    // ...
}

  11. Initialize a GtcrnModel

    main

    To use the GTCRN model for inference, instantiate a GtcrnModel. You can either specify a ModelType explicitly or use the default (which is ModelType::Simple).

    use gtcrn_ladspa_ort::{GtcrnModel, ModelType};

// Create a specific model type
let mut model = GtcrnModel::new(ModelType::Full);

// Or use the default (Simple model)
let mut model_default = GtcrnModel::new_default();

  12. Initialize the StftProcessor for GTCRN

    main

    To perform real-time STFT/iSTFT processing compatible with GTCRN, use StftProcessor::new. For perfect reconstruction with 50% overlap, you must use an nfft of 512 and a hop_size of 256. The processor automatically applies a sqrt(hann) window to both analysis and synthesis stages to ensure mathematical consistency.

    use crate::stft::{StftProcessor, NFFT, HOP_SIZE};

// For GTCRN compatibility:
let mut processor = StftProcessor::new(NFFT, HOP_SIZE);