FullSubNet Documentation

FullSubNet is a full-band and sub-band fusion model designed for real-time single-channel speech enhancement. The repository provides several model variants optimized for different use cases:

• Improved FullSubNet: Optimized to reduce computational costs and support high sampling rates (e.g., 24 KHz and 48 KHz).
• FullSubNet (ICASSP 2021): The original full-band and sub-band fusion model.
• Fast FullSubNet: An accelerated version of the fusion model for single-channel speech enhancement.
• cIRM-based Fullband baseline: A baseline model described in the original FullSubNet paper using complex Ideal Ratio Mask (cIRM).

FullSubNet is a full-band and sub-band fusion model designed for single-channel real-time speech enhancement. It integrates two distinct modeling approaches:

1. Full-band model: Captures global spectral context and long-distance cross-band dependencies, but struggles with signal stationarity and local spectral patterns.
2. Sub-band model: Processes each frequency independently using one target frequency and several context frequencies. It excels at modeling signal stationarity and local spectral patterns but lacks global context.

FullSubNet connects these two models sequentially and uses joint training to leverage their complementary strengths, outperforming many top-ranked methods in the DNS Challenge (INTERSPEECH 2020).

Install the remaining Python packages using pip. Note that Cython should be installed first, and pypesq must be installed directly from its GitHub repository. If your dataset contains .mp3 files, you must also install ffmpeg via Conda.

# Install Cython first
pip install Cython

# Install core libraries
pip install librosa tbb tensorboard joblib pesq pystoi tqdm toml torch_complex rich

# Install pypesq from GitHub
pip install https://github.com/vBaiCai/python-pesq/archive/master.zip

# (Optional) Install ffmpeg for mp3 support
conda install -c conda-forge ffmpeg

To run FullSubNet experiments, your system must meet the following requirements:

• Operating System: Linux-based system (Windows is not supported).
• Hardware: NVIDIA GPU with CUDA and CuDNN installed. CPU-only execution is not supported.
• CUDA Version: CUDA 10.2 or higher is required for GPU training.
• Environment Manager: Anaconda or Miniconda is highly recommended to manage virtual environments and ensure compatible versions of CuDNN and other dependencies.

Clone the repository and navigate into the project directory to begin setup.

git clone https://github.com/haoxiangsnr/FullSubNet
cd FullSubNet

Install the core scientific and deep learning packages via Conda. The project defaults to pytorch=1.12.0. You can choose your preferred CUDA version, but cudatoolkit=11.3 is used in the example below.

# Install PyTorch and CUDA toolkit
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

# Install utility packages
conda install tensorboard joblib matplotlib -c conda-forge

Pre-trained model checkpoints for FullSubNet can be found in the official releases section of the repository.

https://github.com/haoxiangsnr/FullSubNet/releases

Training logs are stored in the directory specified by the save_dir key in your training .toml configuration.

If save_dir is set to ~/Experiments/FullSubNet, the logs will be in ~/Experiments/FullSubNet/train/. This directory contains:

• logs/: TensorBoard data (loss curves, audio files, spectrograms).
• checkpoints/: Model checkpoints for resuming or inference.
• *.toml: A backup of the training configuration.

To visualize the training progress, run TensorBoard pointing to the logs/ directory.

# Basic visualization
tensorboard --logdir ~/Experiments/FullSubNet/train

# Visualize on a specific port
tensorboard --logdir ~/Experiments/FullSubNet/train --port 45454

To reproduce the results reported in the paper, you need Clean Speech and Noise data, as well as Room Impulse Responses (RIRs).

Clean Speech and Noise

The model is evaluated on the Deep Noise Suppression Challenge - INTERSPEECH 2020 (DNS-INTERSPEECH-2020).

• Source: Microsoft DNS-Challenge GitHub
• Note: The default branch of the repository is the ICASSP 2021 dataset. To use the INTERSPEECH 2020 dataset, you must check out the interspeech2020 branch.

Room Impulse Responses (RIRs)

RIRs are sourced from the Multichannel Impulse Response Database and the Reverb Challenge dataset. For convenience, organized RIRs can be downloaded from the FullSubNet Releases page.

Create a dedicated Conda environment named FullSubNet using Python 3.10. If an environment with this name already exists, you should remove it first to avoid conflicts.

# (Optional) remove existing environment
conda env remove --name FullSubNet

# create and activate new environment
conda create --name FullSubNet python=3.10
conda activate FullSubNet