DOCSEARCHbyAlgolia
Submit Docs

OpenSpeech Framework

repository·main·Indexed 20 days ago

https://github.com/openspeech-team/openspeech

A framework for developing end-to-end automatic speech recognition (ASR) models, leveraging PyTorch-Lightning and Hydra for scalable, hardware-agnostic training. It supports a wide range of architectures including Conformer, LAS, RNN-Transducer, and Speech-Transformer, as well as LSTM and Transformer language models. The framework provides tools for training and evaluating acoustic and language models with support for datasets such as LibriSpeech, KsponSpeech, and AISHELL-1.

Tokens
17.2K
Snippets
19
Records
83
Agent score
71%

What's inside OpenSpeech

  1. What is OpenSpeech?

    main

    OpenSpeech is a framework designed for building end-to-end (E2E) automatic speech recognition (ASR) systems. Unlike traditional 'hybrid' ASR systems that require separate training for acoustic, language, and pronunciation models, OpenSpeech focuses on the E2E paradigm which uses a single integrated approach. This simplifies the training pipeline, reduces training and decoding time, and allows for joint optimization with downstream tasks like natural language understanding.

    OpenSpeech is built upon two core libraries:

    • PyTorch-Lightning: Provides hardware-agnostic training, mixed-precision, multi-node, and TPU training support.
    • Hydra: Enables hierarchical configuration management.

    Note: OpenSpeech provides the code and recipes for training models, but it does not provide pre-trained models or APIs for pre-trained models.

  2. Supported Model Architectures in OpenSpeech

    main

    OpenSpeech supports a wide variety of speech recognition and language modeling architectures. While the core concepts of these models are implemented to match their original research, specific implementation details may vary between versions.

    Supported architectures include:

    Sequence-to-Sequence & Attention Models

    • Listen Attend Spell (LAS): Including variations like Location-aware attention, Joint CTC-Attention, Deep CNN Encoder with Joint CTC-Attention, and Multi-head attention based LAS.
    • Speech-Transformer: A no-recurrence sequence-to-sequence model.
    • VGG-Transformer: Transformers with convolutional context.
    • Transformer with CTC: Improving Transformer-based ASR with Connectionist Temporal Classification.
    • Joint CTC-Attention based Transformer

    Transducer & RNN Models

    • DeepSpeech2: End-to-end speech recognition.
    • RNN-Transducer: Sequence transduction with recurrent neural networks.
    • Transformer Transducer: End-to-end speech recognition with self-attention.

    Conformer & Convolutional Models

    • Conformer: Convolution-augmented Transformer, including variants with CTC and LSTM Decoders.
    • Jasper: End-to-end convolutional neural acoustic model.
    • QuartzNet: Deep ASR with 1D time-channel separable convolutions.
    • ContextNet: Improving CNNs for ASR with global context.
    • Squeezeformer: An efficient Transformer for ASR.

    Language Models

    • LSTM Language Model
    • Transformer Language Model

  3. Supported Datasets in OpenSpeech

    main

    OpenSpeech supports the following speech datasets for training and research:

    • LibriSpeech: A corpus of ~1000 hours of 16kHz read English speech derived from LibriVox audiobooks.
    • KsponSpeech: A large-scale spontaneous Korean speech corpus containing 969 hours of open-domain dialog.
      • Note: To use KsponSpeech, you must obtain permission from AI Hub.
    • AISHELL-1: An open-source Chinese Mandarin speech corpus recorded in quiet environments and downsampled to 16kHz.

  4. How Openspeech uses Hydra for configuration

    main

    Openspeech uses Hydra to manage complex, hierarchical configurations. Instead of using pure YAML files, Openspeech utilizes Python @dataclass objects to define module configurations. This approach provides default values directly within the Python code and makes the configuration structure easier to understand and maintain.

    Key benefits include:

    • Hierarchical Composition: Dynamically create configurations by composing different modules (e.g., choosing a specific model, audio type, and trainer).
    • Type Safety: Uses @dataclass (typically inheriting from OpenspeechDataclass) to define parameters.
    • Command Line Overrides: Easily modify any parameter or add new ones via the CLI during runtime.

  5. Available Model Architectures in OpenSpeech

    main

    OpenSpeech implements a wide variety of speech recognition and language modeling architectures. These include:

    Sequence-to-Sequence & Attention Models

    • Listen Attend Spell (LAS): Including Location-aware, Joint CTC-Attention, Deep CNN Encoder, and Multi-head attention variants.
    • Speech-Transformer: A no-recurrence sequence-to-sequence model.
    • VGG-Transformer: Transformers with convolutional context.
    • Conformer: Convolution-augmented Transformers, including variants with CTC and LSTM Decoders.

    Transducer & CTC Models

    • RNN-Transducer
    • Transformer Transducer
    • DeepSpeech2
    • Transformer with CTC
    • Joint CTC-Attention based Transformer

    Convolutional & Hybrid Models

    • Jasper: End-to-end convolutional neural acoustic model.
    • QuartzNet: Uses 1D time-channel separable convolutions.
    • ContextNet: Improving CNNs for ASR with global context.
    • Squeezeformer: An efficient Transformer for ASR.

    Language Models

    • LSTM Language Model
    • Transformer Language Model

    Search Algorithms

    • Beam Search: Available for both RNN Transducer and Transformer Transducer architectures.

  6. How Openspeech uses Hydra for configuration management

    main

    Openspeech utilizes Hydra to manage complex, hierarchical configurations. Instead of using raw YAML files, Openspeech uses Python @dataclass objects (typically inheriting from OpenspeechDataclass) to define module configurations. This approach ensures that default values are stored alongside the module code and provides a clear, type-safe structure for each component.

    Key benefits include:

    • Hierarchical Composition: Build complex configurations by composing smaller, specialized dataclasses.
    • Dynamic Overrides: Modify any parameter via the command line during execution.
    • Plugin Support: Leverages Hydra's ecosystem for hyperparameter sweeping and multi-platform job launching.

  8. Install OpenSpeech

    main

    OpenSpeech requires Python 3.7 or higher. It is recommended to use a virtual environment (venv or conda).

    Prerequisites

    Before installing OpenSpeech, ensure the following dependencies are installed:

    • numpy
    • pytorch (version appropriate for your environment)
    • librosa (via conda install -c conda-forge librosa)
    • torchaudio==0.6.0
    • sentencepiece
    • pytorch-lightning
    • hydra-core
    • warp-rnnt
    • ctcdecode

    Installation Methods

    Via PyPI

    pip install openspeech-core

    From Source

    Run the provided installation script from the repository root:

    ./install.sh

    Install NVIDIA Apex (Optional)

    For faster 16-bit training, install NVIDIA's apex library:

    git clone https://github.com/NVIDIA/apex
cd apex
# Ensure appropriate CUDA and GCC modules are loaded on your cluster
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
    pip install openspeech-core

  9. Train an Acoustic Model with Hydra

    main

    OpenSpeech uses Hydra to manage hierarchical training configurations. You can train various models (e.g., conformer_lstm, listen_attend_spell, quartznet15x5) using the ./openspeech_cli/hydra_train.py script.

    Supported datasets include librispeech, ksponspeech, and aishell.

    Key configuration parameters:

    • dataset: The dataset name.
    • dataset.dataset_path: Path to the dataset.
    • dataset.manifest_file_path: Path to the acoustic model manifest file.
    • tokenizer: The tokenizer type (e.g., libri_subword, kspon_character, aishell_character).
    • model: The model architecture name.
    • audio: Feature type (e.g., fbank, melspectrogram, mfcc).
    • trainer: Hardware target (gpu, tpu, or gpu-fp16).
    python3 ./openspeech_cli/hydra_train.py \
    dataset=librispeech \
    dataset.dataset_download=True \
    dataset.dataset_path=$DATASET_PATH \
    dataset.manifest_file_path=$MANIFEST_FILE_PATH \
    tokenizer=libri_subword \
    model=conformer_lstm \
    audio=fbank \
    lr_scheduler=warmup_reduce_lr_on_plateau \
    trainer=gpu \
    criterion=cross_entropy

  10. Train with hydra_train.py

    main

    To start training, use the hydra_train.py script. You can control the entire training process by passing configuration selections and parameter overrides via the command line.

    Override default values

    Use dot notation to override specific parameters within a module (e.g., model.encoder_dim=320).

    Add new configuration

    To add a parameter that is not part of the default configuration, prefix the key with a + (e.g., +trainer.is_gpu=True).

    # Example: Overriding default values
python ./openspeech_cli/hydra_train.py \
    common=libri \
    common.dataset_path=$DATASET_PATH \
    common.dataset_download=True \
    common.manifest_file_path=$MANIFEST_FILE_PATH \
    vocab=libri_subword \
    vocab.vocab_size=10000 \
    model=conformer_lstm \
    model.encoder_dim=320 \
    audio=mfcc \
    lr_scheduler=warmup_reduce_lr_on_plateau \
    trainer=gpu-fp16 \
    criterion=ctc

# Example: Adding new configuration keys
python ./openspeech_cli/hydra_train.py \
    common=libri \
    vocab=libri_subword \
    model=conformer_lstm \
    audio=mfcc \
    lr_scheduler=warmup_reduce_lr_on_plateau \
    trainer=gpu-fp16 \
    +trainer.is_gpu=True \
    +trainer.is_tpu=False \
    criterion=ctc

  11. Define configuration components using @dataclass

    main

    When creating or migrating components in Openspeech, you should provide a companion Python dataclass. These dataclasses are typically located in the same file as the component and should inherit from OpenspeechDataclass. This allows the component to have its own set of default parameters and metadata.

    Example of defining a configuration dataclass:

    from dataclasses import dataclass, field
from openspeech.dataclass.configurations import OpenspeechDataclass


@dataclass
class ConformerLSTMConfigs(OpenspeechDataclass):
    model_name: str = field(
        default="conformer_lstm", metadata={"help": "Model name"}
    )
    encoder_dim: int = field(
        default=256, metadata={"help": "Dimension of encoder."}
    )

  12. Train a Language Model

    main

    Language model training is performed using ./openspeech_cli/hydra_lm_train.py.

    Requirements:

    • Data must be in a plain text format (one sentence/line per entry).
    • Crucial: You must use the same vocabulary as the acoustic model used for speech recognition.

    Key parameters:

    • dataset.dataset_path: Path to the text file.
    • tokenizer.vocab_path: Path to the vocabulary file (e.g., labels.csv).
    • model: The LM model architecture (e.g., lstm_lm).
    • criterion: The loss function (e.g., perplexity).
    python3 ./openspeech_cli/hydra_lm_train.py \
    dataset=lm \
    dataset.dataset_path=../../../lm.txt \
    tokenizer=kspon_character \
    tokenizer.vocab_path=../../../labels.csv \
    model=lstm_lm \
    lr_scheduler=tri_stage \
    trainer=gpu \
    criterion=perplexity