RawNet Speaker Verification Systems

RawNet2_modified is a refactored PyTorch implementation designed as a baseline for future research. It features a deeper, ResNet-like architecture and improved feature map scaling.

Key Features:

• Architecture: Deeper ResNet-like model with improved feature map scaling ($\alpha$-feature map scaling).
• Loss Function: Uses an Angular loss function.
• Performance: Achieves EER 1.91% (trained using VoxCeleb2, tested on VoxCeleb1 original trial).

RawNet2 is a PyTorch implementation that improves upon the original RawNet by using feature map scaling (similar to squeeze-excitation) to enhance performance.

Key Details:

• Performance: EER 2.56% on VoxCeleb1 original trial.
• Back-end: Uses cosine similarity.
• Core Technique: Feature map scaling for raw waveform speaker verification.

RawNet is a DNN-based speaker embedding extractor designed for use with other DNN-based classifiers. It is implemented in both Keras and PyTorch.

Key Details:

• Performance: EER 4.8% with a cosine similarity back-end, and 4.0% with a proposed concat&mul back-end (VoxCeleb1 original trial).
• Input: Raw waveforms.

The project provides extracted speaker embeddings located in the spk_embd/ directory. These are saved as Python pickle files containing a dictionary. This format is useful for further speaker embedding enhancement or back-end studies.

Data Format:

• File Type: Pickle (.pkl)
• Structure: A dictionary where:
  • Key: Utterance ID (Spk/videoID/segID)
  • Value: Speaker embedding (vector)

To reproduce the exact system described in the Interspeech 2019 paper, use the scripts located in the Keras folder:

1. Environment Setup: Run Keras/lunch_ngc.sh to create a virtual environment for DNN training using NGC.
2. Preprocessing: Use Keras/00-pre_process_waveforms.py to process waveforms.
3. Training: Run Keras/01-trn_RawNet.py to train the model and obtain embeddings (note: minor differences in embeddings may occur due to random seeds).

Run the training process using main.py. You must specify an experiment name (-name) and the model configuration (-module_name and -model_name).

This configuration uses the model_RawNet2 module and the get_RawNet2 model function. When trained on the VoxCeleb2 dataset, this setup achieves an EER of 1.91% on the original VoxCeleb1 trial.

python main.py -name exp1 -module_name model_RawNet2 -model_name get_RawNet2

To use the RawNet2_modified system, you must configure your dataset paths. You can either provide directories via command-line arguments or organize your files into a specific directory structure under a DB/ folder:

• DB/VoxCeleb1: Contains VoxCeleb1 data.
• DB/VoxCeleb2: Contains VoxCeleb2 data.
• DB/augment: Contains musan data.
• DB/augment/RIR_NOISES: Contains RIR data.

Note on musan: The musan dataset must be split to support real-time loading in the Dataloaders. It is recommended to use the script provided in the Voxceleb_trainer repository for this purpose.

RawNet3 is a PyTorch-based implementation for speaker verification using raw waveforms. It supports extracting embeddings from any 16k, 16-bit, mono utterance.

Key Details:

• Performance: Supervised learning with AAM-Softmax achieves EER 0.89%; self-supervised learning achieves EER 5.40%.
• Weights: Pre-trained weights are available on HuggingFace and are included as a submodule in this repository.
• Implementation: PyTorch.
• Benchmark: Vox1-O benchmark results are available in the python/RawNet3 directory.

You can extract RawNet3 embeddings using the ESPnet-SPK framework. This method provides high performance (Vox1-O: EER 0.73%) by leveraging pre-trained models via the Speech2Embedding class.

Prerequisites:

• ESPnet must be installed in your environment.

To use this, replace the placeholder np.zeros with your actual raw waveform data (16k, 16-bit, mono).

import numpy as np 
from espnet2.bin.spk_inference import Speech2Embedding

speech2spk_embed = Speech2Embedding.from_pretrained(model_tag="espnet/voxcelebs12_rawnet3")
embedding = speech2spk_embed(np.zeros(16500)) 　

To extract a speaker embedding from a single audio file, use inference.py with the --inference_utterance flag. The input audio must be in 16k 16bit mono format.

Use the --input flag to specify your file and optionally --out_dir to specify the output path for the .npy embedding file (defaults to ./out.npy).

python inference.py --inference_utterance --input {YOUR_INPUT_FILE} --out_dir {YOUR_OUTPUT_PATH}

To reproduce the Equal Error Rate (EER) reported in the paper, use the pre-trained weights and the provided speaker embeddings for the VoxCeleb1 devset.

Steps:

1. Navigate to the Pre-trained_model folder.
2. Download the extracted RawNet2 speaker embeddings for the VoxCeleb1 devset from this Dropbox link.
3. Place the downloaded speaker embeddings into the spk_embd/ directory.
4. Run the evaluation script evaluate_pretrained_RawNet2.py.

cd Pre-trained_model
python evaluate_pretrained_RawNet2.py

The PyTorch folder contains a baseline implementation using the VoxCeleb2 dataset for training and VoxCeleb1 for validation/evaluation (achieving 3.6% EER on VoxCeleb1).

To run the baseline:

1. Create a virtual environment for DNN training using the NGC (NVIDIA GPU Cloud) setup script: PyTorch/lunch_ngc.sh.
2. Execute the training script train_RawNet.py. Configuration parameters are managed via YAML files.

# 1. Setup environment
./PyTorch/lunch_ngc.sh

# 2. Run training (parameters configured in yaml)
python PyTorch/train_RawNet.py