whisper-diarization

To improve diarization quality, you can isolate vocals from background music using Demucs. This helps the model identify speakers based on speech signals rather than musical characteristics. If the separation fails, the pipeline falls back to the original audio.

if enable_stemming:
    # Isolate vocals from the rest of the audio
    return_code = os.system(
        f'python -m demucs.separate -n htdemucs --two-stems=vocals "{audio_path}" -o "temp_outputs" --device "{device}"'
    )

    if return_code != 0:
        logging.warning("Source splitting failed, using original audio file.")
        vocal_target = audio_path
    else:
        vocal_target = os.path.join(
            "temp_outputs",
            "htdemucs",
            os.path.splitext(os.path.basename(audio_path))[0],
            "vocals.wav",
        )
else:
    vocal_target = audio_path

To use this pipeline, you must install faster-whisper, nemo-toolkit, demucs, deepmultilingualpunctuation, and ctc-forced-aligner. Note that nvidia-cudnn-cu12 should be uninstalled to avoid conflicts.

!pip install "faster-whisper>=1.1.0"
!pip install "nemo-toolkit[asr]>=2.dev"
!pip install git+https://github.com/MahmoudAshraf97/demucs.git
!pip install git+https://github.com/oliverguhr/deepmultilingualpunctuation.git
!pip install git+https://github.com/MahmoudAshraf97/ctc-forced-aligner.git
!pip uninstall -y nvidia-cudnn-cu12

Use the Nvidia NeMo Multi-scale Diarization Decoder (MSDD) to identify different speakers in the audio. The model requires a configuration file (typically a .yaml file) and an input manifest in JSON format.

# Initialize NeMo MSDD diarization model
msdd_model = NeuralDiarizer(cfg=create_config(temp_path)).to("cuda")
msdd_model.diarize()

To fix cases where a sentence is split between two speakers (e.g., Speaker A starts a sentence and Speaker B finishes it with a short interjection), use the get_realigned_ws_mapping_with_punctuation function. It uses a punctuation model to determine the dominant speaker for a sentence based on punctuation marks like ., ?, or !.

if info.language in punct_model_langs:
    # restoring punctuation in the transcript to help realign the sentences
    punct_model = PunctuationModel(model="kredor/punctuate-all")

# ... (punctuation restoration logic)

wsm = get_realigned_ws_mapping_with_punctuation(wsm)
ssm = get_sentences_speaker_mapping(wsm, speaker_ts)

Once processing is complete, you can export the speaker-aware transcript to a plain text file or a SubRip (.srt) subtitle file.

# Export to TXT
with open(f"{os.path.splitext(audio_path)[0]}.txt", "w", encoding="utf-8-sig") as f:
    get_speaker_aware_transcript(ssm, f)

# Export to SRT
with open(f"{os.path.splitext(audio_path)[0]}.srt", "w", encoding="utf-8-sig") as srt:
    write_srt(ssm, srt)

Users should be aware of the following current limitations:

• Overlapping Speakers: The pipeline does not yet address overlapping speakers. A potential future approach involves isolating single speakers via audio separation, though this increases computational cost.
• General Errors: Users are encouraged to raise issues if errors are encountered.

To process an audio file, use the diarize.py script.

If your system has at least 10GB of VRAM, you can use diarize_parallel.py instead. This version runs NeMo in parallel with Whisper to speed up processing, though it is currently considered experimental.

The following flags are available for the diarization CLI: