DOCSEARCHbyAlgolia
Submit Docs

MossFormer2 Speech Separation

repository·main·Indexed 41 days ago

https://github.com/alibabasglab/mossformer2

A monaural speech separation model that integrates Transformer modules with RNN-free recurrent networks using Feedforward Sequential Memory Networks (FSMN). It is designed to model long-range coarse-scale dependencies and fine-scale recurrent patterns. The model can be used via the ModelScope pipeline API for inference with 8000Hz single-channel WAV files, or through a standalone version for minimal dependency inference.

Tokens
879
Snippets
3
Records
4
Agent score
2%

What's inside MossFormer2

  1. Overview of MossFormer2

    main

    MossFormer2 is a monaural speech separation model that combines a Transformer-based module with an RNN-free recurrent module. It is designed to model both long-range, coarse-scale dependencies (via self-attention) and fine-scale recurrent patterns (via an enhanced dilated FSMN block). This hybrid approach allows for more effective speech separation compared to previous models like MossFormer.

    Key features include:

    • Transformer Module: Uses full-computation self-attention on local segments and linearized self-attention for the entire sequence.
    • Recurrent Module: An 'RNN-free' module based on a Feedforward Sequential Memory Network (FSMN) that uses gated convolutional units (GCU) and dense connections to capture recurrent patterns without traditional recurrent connections.

  2. Use the MossFormer2 standalone model for inference

    main

    The MossFormer2_standalone directory provides a version of the MossFormer2 model designed for inference with a minimal set of dependencies. This version is intended for running pre-trained models rather than training them. For training, you should use Speechbrain. To implement inference in your own code, refer to the sample code provided in inference.py.

    # Refer to inference.py for sample inference code.

  3. Install MossFormer2 dependencies

    main

    MossFormer2 is used via the ModelScope framework. To perform inference, you must have modelscope installed. The pipeline also relies on SoundFile for WAV file processing.

    Platform-specific requirements for libsndfile:

    • Linux (Ubuntu example): You must manually install the libsndfile1 dependency.
    • Windows/MacOS: The dependency is installed automatically.

    To install the required system dependency on Ubuntu, run:

    sudo apt-get update
sudo apt-get install libsndfile1

  4. Perform monaural speech separation with MossFormer2

    main

    You can use the speech_mossformer2_separation_temporal_8k model via the ModelScope pipeline API.

    Input Requirements:

    • A single-channel WAV file.
    • Sample rate must be 8000Hz.
    • The file should contain mixed speech from two people.

    Output:

    • The pipeline returns a list of separated single-channel audio signals in output_pcm_list.
    • You can provide either a local file path or a URL as the input.
    import numpy
import soundfile as sf
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

# input can be a URL or a local path
input = 'https://modelscope.cn/api/v1/models/damo/speech_mossformer2_separation_temporal_8k/repo?Revision=master&FilePath=examples/mix_speech1.wav'
separation = pipeline(
   Tasks.speech_separation,
   model='damo/speech_mossformer2_separation_temporal_8k')
result = separation(input)
for i, signal in enumerate(result['output_pcm_list']):
    save_file = f'output_spk{i}.wav'
    sf.write(save_file, numpy.frombuffer(signal, dtype=numpy.int16), 8000)