3D-Speaker Toolkit

This section details the training configuration and performance metrics for the Res2Net model used for speaker verification on the Voxceleb dataset.

Training Configuration

• Features: 80-dim fbank, mean normalization, and speed perturbation.
• Hyperparameters:
  • Learning rate (lr): [0.00005, 0.2]
  • Batch size: 512
  • Hardware: 8 GPUs (Tesla V100)
  • Loss: Additive angular margin
• Evaluation Metrics: EER (%) and MinDCF (p-target=0.01).

Voxceleb Performance Results

• Training Set: Voxceleb2-dev (5994 speakers)
• Test Set: Voxceleb-O

This section details the configuration and performance metrics for the ResNet-based speaker verification model trained on the Voxceleb dataset.

Training Configuration

• Features: 80-dim fbank, mean normalization, and speed perturbation.
• Hyperparameters: Learning rate (lr) in range [0.00005, 0.2], batch size of 256.
• Hardware: Optimized for 4 GPUs (Tesla V100).
• Loss Function: Additive angular margin.
• Evaluation Metrics: EER (%) and MinDCF (p-target=0.01).

Voxceleb Performance Results

• Training Set: Voxceleb2-dev (5994 speakers).
• Test Set: Voxceleb-O.

To use the speaker diarization tools, install the required Python packages using the provided requirements file.

pip install -r requirements.txt

Execute the Dialogue Detection task using the provided shell script. The script requires an output directory as the only parameter.

Command: bash run_dialogue_detection.sh <output_dir>

Core Python logic: egs/semantic_speaker/bert/bin/run_dialogue_detection.py

Key Arguments for run_dialogue_detection.py:

• --model_name_or_path: Path to pre-trained BERT model (e.g., bert-base-chinese).
• --train_file, --validation_file, --test_file: Paths to the JSON files for training, validation, and testing.

3D-Speaker supports advanced diarization and language identification tasks:

• Speaker Diarization: Includes standard audio-based diarization and multimodal diarization which fuses audio and video input for improved accuracy. It also supports Dialogue Detection and Semantic Speaker Change Detection.
• Language Identification: Models that integrate phonetic information for higher recognition accuracy, supporting both Mandarin and English (e.g., using CAM++ or ERes2Net).

Recipes can be found in egs/3dspeaker/speaker-diarization and egs/3dspeaker/language-identification.

Execute the Speaker-Turn Detection task using the provided shell script. The script requires an output directory as the only parameter.

Command: bash run_speaker_turn_detection.sh <output_dir>

Core Python logic: egs/semantic_speaker/bert/bin/run_speaker_turn_detection.py

Key Arguments for run_speaker_turn_detection.py:

• --model_name_or_path: Path to pre-trained BERT model.
• --train_file, --validation_file, --test_file: Paths to the JSON files.
• --text_column_name: Column name for text (defaults to sentence).
• --label_column_name: Column name for labels (defaults to change_point_list).
• --label_num: Number of labels.

bash run_speaker_turn_detection.sh exp/

This recipe implements Active Speaker Detection using the TalkNet model, adapted for multi-process data processing and multi-GPU training/evaluation.

To use this recipe, install the required Python dependencies and execute the run script. You must also ensure ffmpeg is installed in your environment.

pip install -r requirements.txt
bash run.sh

The toolkit provides recipes and pretrained models for several speaker verification architectures:

• ERes2NetV2: High-performance models (e.g., speech_eres2netv2_sv_zh-cn_16k-common) trained on large datasets.
• CAM++: Efficient speaker verification models.
• ECAPA-TDNN: Standard speaker verification architecture.

Recipes for training and inference are available in the egs/ directory for various datasets like 3D-Speaker, VoxCeleb, and CN-Celeb.

You can run training or evaluation experiments for various speaker verification models on the CN-Celeb dataset by navigating to the specific model directory and executing the run.sh script. Supported models include ERes2Net, CAM++, ECAPA-TDNN, ResNet, Res2Net, and RDINO (self-supervised).

# Speaker verification: ERes2Net on CN-Celeb
cd egs/cnceleb/sv-eres2net/
bash run.sh

# Speaker verification: CAM++ on CN-Celeb
cd egs/cnceleb/sv-cam++/
bash run.sh

# Speaker verification: ECAPA-TDNN on CN-Celeb
cd egs/cnceleb/sv-ecapa/
bash run.sh

# Speaker verification: ResNet on CN-Celeb
cd egs/cnceleb/sv-resnet/
bash run.sh

# Speaker verification: Res2Net on CN-Celeb
cd egs/cnceleb/sv-res2net/
bash run.sh

# Self-supervised speaker verification: RDINO on CN-Celeb
cd egs/cnceleb/sv-rdino/
bash run.sh