Awesome Speaker Diarization

Awesome Speaker Diarization is a curated collection of resources dedicated to the field of speaker diarization. It provides a centralized repository of research papers, software libraries, datasets, and learning materials to help developers and researchers implement or study speaker diarization technologies.

The resources are organized into several key categories:

• Publications: Research papers and academic findings.
• Software: Categorized into frameworks, evaluation tools, clustering algorithms, speaker embedding, speaker change detection, audio feature extraction, and data augmentation.
• Datasets: Including diarization datasets, speaker embedding training sets, and augmentation noise sources.
• Learning Materials: Online courses, books, tech blogs, and video tutorials.
• Products: Commercial or ready-to-use diarization products.

Use these toolkits to perform reproducible evaluation, diagnostic analysis, and error calculation for diarization systems:

• pyannote-metrics (Python): Toolkit for reproducible evaluation and error analysis.
• SimpleDER (Python): Lightweight library to compute Diarization Error Rate (DER).
• DiarizationLM (Python): Implements Word Error Rate (WER), Word Diarization Error Rate (WDER), and concatenated minimum-permutation Word Error Rate (cpWER).
• dscore (Python & Perl): Diarization scoring tools.
• spyder (Python & C++): Fast DER computation.
• CDER (Python): Conversational DER for the CSSD task.

The repository provides various educational resources for developers learning diarization:

• Online Courses: Udemy tutorials.
• Books: 'Voice Identity Techniques' by Quan Wang.
• Tech Blogs: Articles covering literature reviews, Kaldi integration, and building modules from scratch.
• Video Tutorials: Deep dives into pyannote.audio, Google's LSTM systems, and optimal clustering techniques.

These papers document specific challenges and competitive benchmarks in the field:

• M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge (2022)
• Hitachi-JHU DIHARD III system: Competitive end-to-end neural diarization and x-vector clustering systems combined by DOVER-Lap (2021)
• DIHARD Challenge Lessons: Diarization is Hard: Some Experiences and Lessons Learned for the JHU Team (2018)
• ODESSA: Albayzin Speaker Diarization Challenge 2018 (2018)
• DIHARD Challenge: Joint Discriminative Embedding Learning, Speech Activity and Overlap Detection (2018)

If you require production-ready APIs or applications rather than building your own, the repository lists several products:

• Cloud APIs: Google Cloud Speech-to-Text, Amazon Transcribe, IBM Watson Speech To Text, Microsoft Azure Conversation Transcription, and DeepAffects API.
• Consumer/Desktop Apps: Google Recorder, Alibaba Tingwu, Kviskr (macOS), and Ariso AI's OATS (macOS).

These papers focus on real-time or online diarization approaches, including bandit learning and relation networks:

• MiniVox: Speaker Diarization as a Fully Online Bandit Learning Problem (2021)
• Relation Network: Online Speaker Diarization with Relation Network (2020)
• VoiceID on the Fly: A Speaker Recognition System that Learns from Scratch (2020)

These papers investigate the integration of speaker diarization with Automatic Speech Recognition (ASR), including speaker-attributed ASR and joint sequence transduction:

• Comparative Study: Speaker-attributed Automatic Speech Recognition in Multi-party Meetings (2022)
• Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection (2021)
• Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASR (2021)
• Joint Speech Recognition and Speaker Diarization via Sequence Transduction (2019)
• Deep learning models for joint speech recognition, segmentation and diarization (2018)

Tools for processing audio signals and generating training data:

Feature Extraction:

• LibROSA (Python): Library for audio and music analysis.
• python_speech_features (Python): Provides MFCCs and filterbank energies.
• pyAudioAnalysis (Python): Feature extraction, classification, and segmentation.

Data Augmentation:

• pyroomacoustics (Python): Signal processing for indoor applications (beamforming prototyping).
• gpuRIR (Python): Room Impulse Response (RIR) simulation with GPU acceleration.
• rir_simulator_python (Python): Python-based RIR simulator.
• WavAugment (Python & PyTorch): Performs augmentation on PyTorch tensors.
• EEND_dataprep (Bash & Python): Recipes for generating simulated conversations for EEND training.