ClearerVoice-Studio

SpeechScore is a toolkit for both intrusive and non-intrusive speech quality assessment.

Intrusive Metrics (Require reference audio):

• BSSEval {ISR, SAR, SDR}
• {CBAK, COVL, CSIG}
• FWSEGSNR
• LLR
• LSD
• MCD
• NB_PESQ
• PESQ
• SISDR
• SNR
• SSNR
• STOI

Non-Intrusive Metrics (No reference audio required):

• DNSMOS {BAK, OVRL, SIG, P808_MOS}
• SRMR
• NISQA
• DISTILL_MOS

ClearerVoice-Studio is an AI-powered speech processing toolkit providing the following core capabilities:

• Speech Enhancement: Denoising and improving speech quality.
• Speech Separation: Separating different speakers or audio sources.
• Speech Super-Resolution: Also known as bandwidth extension; converts low-resolution audio (min. 16,000 Hz) into high-resolution audio (48,000 Hz).
• Target Speaker Extraction: Extracting specific speakers using various conditions (audio-only, audio-visual via face/lips, body gestures, or EEG signals).
• Speech Quality Assessment: Using the SpeechScore toolkit to evaluate models with metrics like SNR, PESQ, STOI, DNSMOS, SI-SDR, NISQA, and DISTILL_MOS.

This task extracts a target speaker conditioned on body gestures.

• Supported Model: SEG (Non-causal)
• Supported Dataset: YGD

This task extracts a target speaker conditioned on visual cues from face or lip recordings.

• Supported Models:
  • AV-ConvTasNet (Causal/Non-causal)
  • AV-DPRNN (aka USEV) (Non-causal)
  • AV-TFGridNet (Non-causal)
  • AV-TFGridNet-ISAM (Non-causal)
  • AV-Mossformer2 (Non-causal)
• Supported Datasets:
  • VoxCeleb2
  • LRS2

This task involves extracting a target speaker conditioned on a reference speech signal.

• Supported Model: SpEx+ (Non-causal)
• Supported Dataset: WSJ0-2mix
• Configuration Example: config/config_wsj0-2mix_speech_SpEx-plus_2spk.yaml

This task extracts a target speaker conditioned on EEG signals.

• Supported Model: NeuroHeed (Non-causal)
• Supported Dataset: KUL

Execute the training process using the train.sh script. You must configure the network type, whether to resume from a checkpoint, and the initial checkpoint path within train.sh before running.

To start training, run:

bash train.sh

Configuration Variables in train.sh:

• network: Specify the model architecture (e.g., MossFormer2_SE_48K).
• train_from_last_checkpoint: Set to 1 to resume training from the most recent checkpoint if one exists.
• init_checkpoint_path: The path to your initial model if performing fine-tuning; set to None if starting fresh.

To use the speech separation training scripts, clone the repository and create a dedicated Conda environment with Python 3.8 and the required dependencies.

git clone https://github.com/modelscope/ClearerVoice-Studio.git
cd ClearerVoice-Studio
conda create -n ClearerVoice-Studio python=3.8
conda activate ClearerVoice-Studio
pip install -r requirements.txt

To use the reverb and noise generation script, follow these steps:

1. Prepare file lists: Create two .scp files containing paths to your datasets:
   • A clean speech file list (e.g., data/data_scp/speech.scp).
   • A noise file list (e.g., data/data_scp/noise.scp).
2. Configure settings: Modify the parameters in config/para.cfg to suit your specific training requirements.
3. Run a test: Execute the provided demo script to verify the setup:

bash run.sh

4. Loop generation (Optional): To generate multiple different outputs, set the run_num variable in run.sh to your desired number of iterations.

To use SpeechScore, clone the repository and set up a Conda environment with Python 3.8.

# Clone the repository
git clone https://github.com/modelscope/ClearerVoice-Studio.git

# Create and activate Conda environment
cd ClearerVoice-Studio
conda create -n ClearerVoice-Studio python=3.8
conda activate ClearerVoice-Studio

# Install dependencies
pip install -r requirements.txt

The repository provides training and fine-tuning scripts for several tasks:

• Speech Enhancement: Supports 16kHz and 48kHz.
• Speech Separation: Supports 8kHz and 16kHz.
• Speech Super-Resolution: Supports 48kHz.
• Target Speaker Extraction:
  • Audio-only (conditioned on reference speech, 8kHz)
  • Audio-visual (conditioned on face/lips, 16kHz)
  • Audio-visual (conditioned on body gestures, 16kHz)
  • Neuro-steered (conditioned on EEG signals, 16kHz)

Training requires two .scp files that list the training and testing data.

1. Create your training data list (e.g., data/tr_demand_28_spks_16k.scp).
2. Create your testing data list (e.g., data/cv_demand_testset_16k.scp).
3. Update the paths to these .scp files in your configuration files located in config/train/*.yaml.

Recommendation: If your training dataset is small (<100 hours) and utterances are longer than 20 seconds, crop them into smaller segments (4s to 5s) to increase data size and reduce loading time.