DOCSEARCHbyAlgolia
Submit Docs

NVIDIA NeMo Speech Toolkit

repository·main·Indexed 30 days ago

https://github.com/nvidia-nemo/speech

A comprehensive framework for Speech AI providing tools and pre-trained models for Automatic Speech Recognition (ASR), Text-to-Speech (TTS), Speech LLMs, and audio processing. Includes support for specialized tasks like speaker diarization, voice activity detection (VAD), speech enhancement, and multi-channel audio processing. Features NeMo Lightning for PyTorch Lightning-compatible training of NeMo 2.0 models using Megatron Core.

Tokens
239K
Snippets
422
Records
949
Agent score
95%

What's inside nvidia-nemo-speech

  1. Overview of SpeechLM2 model types

    main

    SpeechLM2 is a collection that augments pre-trained Large Language Models (LLMs) with speech understanding and generation. Supported model types include:

    • SALM (Speech-Augmented Language Model): Augments LLMs with speech understanding.
      • SALM: Uses HuggingFace Transformers backbone with optional HF PEFT LoRA.
      • SALMAutomodel: Uses NeMo Automodel backbone with native LoRA, advanced parallelism (FSDP2, TP, SP, EP), and MoE optimizations (Grouped GEMM, DeepEP).
    • DuplexS2SModel: Full-duplex speech-to-speech model with an ASR encoder predicting discrete audio codes.
    • DuplexS2SSpeechDecoderModel: A variant of DuplexS2SModel with a separate transformer decoder for speech generation.
    • DuplexEARTTS: Duplex text-to-speech model supporting user interruption via a special text interruption token.
    • DuplexSTTModel: Decoder model that generates agent text in duplex mode in response to user speech and text.
    • NemotronVoiceChat: An inference-only pipeline that chains DuplexSTTModel and DuplexEARTTS for end-to-end conversational agents.

  2. Explore the NeMo Audio Collection for speech enhancement and restoration

    main

    The NeMo Audio Collection provides models for various audio processing tasks, specifically focusing on speech enhancement, restoration, and multi-channel processing.

    Supported task categories include:

    • Mask-based speech processing: Includes single-channel masking and Guided Source Separation (GSS).
    • Predictive speech processing: Utilizes NCSN++.
    • Generative models: Includes Score-based generative models (SGMSE+) and Schrödinger bridge-based models.
    • Flow-matching-based models.
    • Multi-channel audio processing: Includes mask-based beamforming (MVDR) and dereverberation (WPE).

    For detailed implementation guides and API references, refer to the official NeMo documentation.

  3. Explore NeMo TTS model recipes, configurations, and checkpoints

    main

    NeMo provides several resources for working with Text-to-Speech models:

    • Model Recipes: Python scripts for model training and usage can be found in examples/tts/*.py.
    • Configuration Files: YAML configuration files for TTS models are located in examples/tts/conf/.
    • Pretrained Model Checkpoints: Ready-to-use checkpoints are available for immediate speech synthesis or fine-tuning on custom datasets. Refer to the checkpoints documentation for specific usage instructions.

  4. Explore Parakeet ASR model families

    main

    Parakeet is a family of ASR models utilizing a FastConformer Encoder with CTC, RNN-T, or TDT decoders. Key variants include:

    • Parakeet-TDT-0.6B V3: 25 languages, supports PnC (Punctuation and Capitalization), high speed.
    • Parakeet-TDT-0.6B V2: English-only, supports PnC, high speed.
    • Parakeet-TDT/CTC-110M: Optimized for edge deployment.
    • Nemotron-3.5-ASR-Streaming: Real-time streaming support for 40 languages.
    • Multitalker-Parakeet: Designed for multi-speaker streaming scenarios.

  5. Explore Speaker Recognition models

    main

    NeMo provides several pre-trained model architectures for speaker recognition tasks, including speaker verification and diarization. You can use these checkpoints for immediate inference or fine-tune them on your own datasets.

    Available model architectures:

    • TitaNet: Based on the ContextNet architecture, using 1D depth-wise separable convolutions with Squeeze-and-Excitation (SE) layers and channel attention-based statistics pooling to map variable-length utterances to fixed-length embeddings (tvectors).
    • SpeakerNet: Based on the QuartzNet ASR architecture. It uses an encoder-decoder structure where the encoder extracts features and a statistics pooling layer computes mean and variance to capture time-independent speaker features. It provides different sizes (e.g., SpeakerNet-L and SpeakerNet-M) with different embedding sizes.
    • ECAPA_TDNN: Based on Emphasized Channel Attention, Propagation, and Aggregation. It uses time dilation layers, Multi-layer Feature Aggregation (MFA), and Squeeze-Excitation (SE) blocks.

  6. Understand SpeechLM2 Dataset Formats

    main

    The speechlm2 collection uses the Lhotse framework to manage datasets containing audio and text. Depending on your model type, you will use one of the following dataset classes:

    • DuplexS2SDataset: For general duplex speech-to-speech models (speech input $\rightarrow$ speech output).
    • SALMDataset: For Speech-Augmented Language Models (speech+text input $\rightarrow$ text output).
    • DuplexSTTDataset: For models processing conversational audio to generate text responses.
    • DuplexEARTTSDataset: For Duplex EARTTS models, extending DuplexS2SDataset with additional fields for TTS, such as audio_prompt and context_audio for speaker conditioning.

    Datasets are organized around conversation turns. Each turn contains audio and text associated with a specific speaker role (e.g., 'user' vs 'assistant').

  7. Use Speech Data Processor (SDP) for dataset processing

    main

    Speech Data Processor (SDP) is a specialized toolkit designed to streamline speech dataset preparation. It helps developers:

    1. Minimize boilerplate: Write less code when implementing processing logic for new datasets.
    2. Share workflows: Standardize and share the specific steps used to process a speech dataset.

    Note that SDP is a separate repository from the main NeMo toolkit. You can find the source code and additional details at https://github.com/NVIDIA/NeMo-speech-data-processor and the full documentation at https://nvidia.github.io/NeMo-speech-data-processor/.

  8. Explore Automatic Speech Recognition (ASR) features and models

    main

    NVIDIA NeMo provides a wide range of ASR capabilities, including transcription, streaming, and specialized speech tasks.

    Supported ASR Model Architectures

    • Jasper, QuartzNet, CitriNet, ContextNet
    • Conformer-based: Conformer-CTC, Conformer-Transducer, FastConformer-CTC, FastConformer-Transducer
    • Squeezeformer-based: Squeezeformer-CTC, Squeezeformer-Transducer
    • LSTM-based: LSTM-Transducer (RNNT), LSTM-CTC

    Supported Decoders and Losses

    • CTC
    • Transducer/RNNT
    • Hybrid Transducer/CTC
    • NeMo Original: Multi-blank Transducers and Token-and-Duration Transducers (TDT)

    Specialized Speech Tasks

    • Streaming/Buffered ASR: Supports chunked inference for low-latency applications.
    • Cache-aware Streaming Conformer: Optimized for microphone streaming with multiple lookaheads.
    • Language Modeling: N-gram LM fusion with Beam Search decoding, or Neural Rescoring with Transformer.
    • Speech Classification: MatchboxNet (Command Recognition) and AmberNet (Language Identification).
    • Voice Activity Detection (VAD): Uses MarbleNet.
    • Speaker Recognition: TitaNet, ECAPA_TDNN, and SpeakerNet.
    • Speaker Diarization: Clustering Diarizer (TitaNet, ECAPA_TDNN, SpeakerNet) and Neural Diarizer (Sortformer).
    • Speech Intent Detection and Slot Filling: Conformer-Transformer architecture.

  9. Understand the SpeechLM2 collection structure

    main

    The speechlm2 collection is organized into three main components:

    • Models: Implementations of DuplexS2SModel, DuplexS2SSpeechDecoderModel, DuplexSTTModel, SALM, SALMAutomodel, DuplexEARTTS, and the inference-only NemotronVoiceChat.
    • Modules: Contains audio perception and speech generation modules.
    • Data: Includes dataset classes and data loading utilities.

  10. Understand Speaker Recognition (SR) in NeMo

    main

    Speaker Recognition (SR) in NeMo focuses on text-independent speaker recognition, where identity is determined by how speech is spoken rather than the content. The system converts unconstrained speech utterances into fixed-length vectors known as speaker embeddings.

    NeMo supports two primary SR tasks:

    1. Speaker Identification: Determining the identity of a speaker (e.g., "Who is speaking?"). This is typically achieved by training a model with cross-entropy loss and fine-tuning it on specific sets of known speakers.
    2. Speaker Verification: Confirming if a speaker is who they claim to be (e.g., "Is this speaker User A?"). This involves training an embedding extractor using angular softmax loss and comparing embeddings from different audio files using scoring techniques like cosine similarity.

    Speaker embeddings generated by these models can also be utilized in downstream tasks such as Automatic Speech Recognition (ASR) and Speech Synthesis.

  11. Download speaker tasks datasets for diarization

    main

    Use the scripts located in scripts/dataset_processing/speaker_tasks/ to download datasets specifically designed for speaker tasks, such as speaker diarization. These scripts automatically convert the downloaded data into NeMo-compatible manifest files, which can be used directly with NeMo Diarization models.

    Note: While scripts exist for CallHome and DIHARD3 datasets, the actual data for these must be downloaded separately. If you need the specific scripts for these datasets, you should open an issue in the repository.

  12. Use Speech Classification models for Commands, VAD, and Lang ID

    main

    NeMo's Speech Classification collection supports three primary tasks:

    1. Speech Command (Keyword) Detection: Uses the MatchboxNet family of models. These are end-to-end neural networks optimized for high accuracy with low parameter counts. Models are denoted as MatchBoxNet_[BxRxC] (Blocks, sub-blocks, channels). Versions like _v1 or _v2 indicate the dataset used (e.g., 30-way or 35-way classification).
    2. Voice Activity Detection (VAD): Uses the MarbleNet family of models, which are based on the MatchboxNet architecture but optimized for VAD tasks (e.g., on the AVA speech dataset).
    3. Language Identification (Lang ID): Uses the AmberNet model, which is based on the TitaNet architecture and optimized for datasets like Voxlingua107.

    To use these models with existing ASR scripts, specify the corresponding model architecture in your configuration file. Example configuration files are located in <NeMo_git_root>/examples/asr/conf.