WeSpeaker Toolkit

The VoxConverse v1 recipe is designed for speaker diarization tasks using the Voxconverse2020 dataset (dev set: 216 utts).

Key Components:

• Speaker Model: ResNet34 model (pretrained by WeSpeaker). You can find the pretrained model at: https://wespeaker-1256283475.cos.ap-shanghai.myqcloud.com/models/voxceleb/voxceleb_resnet34_LM.onnx.
• Speaker Activity Detection (SAD): Supports both oracle SAD (ground truth) and system SAD (using silero-vad).
• Clustering: Uses spectral clustering.
• Metric: Performance is measured using DER (Diarization Error Rate), which is the sum of MISS, FALSE ALARM, and SPEAKER CONFUSION percentages.

Requirements:

• A machine with GPU availability.
• onnxruntime-gpu support.

The Voxconverse recipe provides speaker diarization for the Voxconverse 2020 dataset, which focuses on 'in the wild' scenarios from YouTube videos.

There are two versions of the recipe available:

• v1: The original monolithic recipe.
• v2: A modular recipe that splits the process into distinct stages: Fbank extraction, embedding extraction, and clustering.

Recommendation: Newcomers should use the v2 recipe and execute it stage by stage for better control and modularity.

WeSpeaker provides several specialized tutorials and guides for different tasks including supervised speaker verification, self-supervised learning, and diarization.

Key resources include:

• Supervised Speaker Verification (SV): Tutorial on VoxCeleb v2.
• Self-Supervised Speaker Verification (SSL): Tutorial on VoxCeleb v3 using DINO.
• Speaker Diarization: Tutorial on VoxConverse v2.
• Pretrained Models: A list of available off-the-shelf models.
• Usage Guides: Instructions for using WeSpeaker via the Command Line or Python API.
• Research: Suggested papers for speaker embedding learning.

WeSpeaker supports self-supervised speaker verification using the SimCLR framework. This approach is based on contrastive learning techniques for text-independent speaker verification.

Key configuration details for the provided results include:

• Features: fbank80
• Frame count: num_frms200
• Training: epoch150, InfoNCE loss, aug_prob1.0
• Augmentation: speed_perturb, no spec_aug
• Scoring: cosine (sub mean of vox2_dev)
• Metric: EER(%)

WeSpeaker provides two types of pretrained models for speaker verification (SV) and speaker embedding extraction:

1. Checkpoint Model (.pt suffix): These are standard PyTorch checkpoints. They are used to reproduce published results or to serve as a starting point for continuing training.
2. Runtime Model (.onnx suffix): These models are exported via Onnxruntime from the checkpoint models and are intended for efficient inference.

Speaker embeddings can be used for tasks such as voice conversion, text-to-speech, speaker adaptive ASR, and target speaker extraction.

The CNC-Eval v2 recipe was updated (2022.07.12) to align with the winning system of CNSRC 2022. Key architectural and training changes include:

• Implementing an LR scheduler warmup starting from 0.
• Removing one embedding layer.
• Adding a large margin fine-tuning (LM) strategy.

These changes significantly improve performance across various models (ResNet, ECAPA-TDNN, RepVGG) when combined with AS-Norm and LM strategies.

WeSpeaker supports downloading default models or using custom ones.

Default Models: Specifying the --language option will automatically download:

• english: ResNet221_LM (pretrained on VoxCeleb)
• chinese: ResNet34_LM (pretrained on CnCeleb)

Custom Models: To use a custom model, use the --pretrain flag to point to a directory. Important: You must rename your model and config files to avg_model.pt and config.yaml respectively within that directory.

WeSpeaker supports various training pipelines and is evaluated/trained on several large-scale datasets:

Pipelines

• DINO Pretraining: Leveraging in-the-wild data for effective self-supervised pretraining in speaker recognition.

Datasets

• VoxBlink / VoxBlink2: Large-scale speaker verification datasets (including camera-based and 100K+ speaker corpora).
• VoxCeleb / VoxCeleb2: Standard large-scale speaker identification datasets.
• CNCeleb: A challenging Chinese speaker recognition dataset.

WeSpeaker supports self-supervised speaker verification using the MoCo (Momentum Contrast) framework. This approach allows for unsupervised representation learning.

As of the July 2023 update, the implementation is compatible with models like ECAPA_TDNN_GLOB_c512-ASTP-emb192 trained using the MoCo method.

Training/Setup Configuration used in benchmarks:

• Features: fbank80
• Frames: num_frms200
• Epochs: epoch150
• Loss: InfoNCE
• Augmentation: aug_prob1.0, speed_perturb, no spec_aug
• Scoring: cosine (sub mean of vox2_dev)
• Metric: EER(%)