DOCSEARCHbyAlgolia
Submit Docs

Mozilla DeepSpeech

repository·master·Indexed 12 days ago

https://github.com/mozilla/deepspeech

An open-source speech-to-text engine. This documentation includes guides for building dependencies like OpenFST and kenlm, using the kenlm Python API for language model scoring, and utilizing utility scripts for dataset creation (data_set_tool.py), audio sample comparison (compare_samples.py), and importing the Aidatatang corpus.

Tokens
61.4K
Snippets
230
Records
324
Agent score
97%

What's inside DeepSpeech

  1. Identify language-specific data files

    master

    The data/ directory contains essential files for language-specific processing in DeepSpeech:

    1. Alphabet File: data/alphabet.txt contains the list of unique characters for the target language.
    2. Language Model Generation Script: data/lm/generate_lm.py is used to generate a binary n-gram language model.

    If you need to create an alphabet file from your own training CSV files, use the check_characters utility from the training package.

    # To see how to create an alphabet file from training CSVs:
python -m deepspeech_training.util.check_characters --help

  2. Project DeepSpeech Overview

    master

    DeepSpeech is an open-source Speech-To-Text engine based on Baidu's Deep Speech research. It utilizes Google's TensorFlow for its implementation.

    Note: This project is now discontinued.

    For detailed information on installation, usage, and training models, refer to the official documentation at deepspeech.readthedocs.io.

  3. Configure augmentation parameter ranges

    master

    When specifying parameters for augmentations, you can use constant values or dynamic ranges to vary the augmentation intensity over the course of training:

    • <value>: A constant (int or float).
    • <value>~<r>: A center value with a randomization radius. E.g., 1.2~0.4 picks a random value between 0.8 and 1.6.
    • <start>:<end>: A value that ranges from <start> at the beginning of training to <end> at the end of training.
    • <start>:<end>~<r>: A combination where the center value ranges from <start> to <end>, with a randomization radius of <r> applied at each step.

  4. How the DeepSpeech scorer is composed

    master

    A DeepSpeech scorer package is composed of two sub-components:

    1. A KenLM language model (generated via data/lm/generate_lm.py).
    2. A trie data structure containing all words in the vocabulary.

    To create a complete scorer package, you must first generate the KenLM language model and then use the generate_scorer_package binary to combine it with the vocabulary into a final package file.

  6. Enable TensorFlow Lite Delegation on Android

    master

    DeepSpeech Android builds support experimental delegation to offload computation from the CPU to other hardware via the TensorFlow Lite Delegate API.

    Supported delegates:

    • gpu: Leverages OpenGL capabilities.
    • nnapi: Uses the Android API to leverage GPU, DSP, or NPU.
    • hexagon: Leverages Qualcomm-specific DSP.

    How to use: Set the DS_TFLITE_DELEGATE environment variable to one of the values above (only one at a time).

    Warnings:

    • This is highly experimental.
    • You may need to use exported models that support these delegates.
    • Performance gains are not guaranteed.
    # Example: Enabling GPU delegation
export DS_TFLITE_DELEGATE=gpu

  7. Choose between Alphabet-based and Bytes output modes

    master

    DeepSpeech operates in two distinct decoding modes. Crucially, you must match the model type with the correct scorer type.

    1. Default Mode (Alphabet-based)

    • Mechanism: Uses an alphabet file (specified via --alphabet_config_path during training/export) to determine predicted labels.
    • Scorer Requirement: If using an external scorer, it MUST be word-based and built using the same alphabet file used for training. Words in the scorer's text corpus must be separated by whitespace.

    2. Bytes Output Mode

    • Mechanism: Predicts UTF-8 bytes directly instead of characters from an alphabet. Enabled via the --bytes_output_mode flag during training/export.
    • Labels: The model has 256 labels (0-254 for UTF-8 byte values, 255 for the CTC blank symbol).
    • Scorer Requirement: If using an external scorer, it MUST be a UTF-8 (character-based) scorer.
    • Use Case: Useful for large alphabets (e.g., Mandarin) or multi-language models.

    Compatibility Warning

    • Acoustic models trained with --bytes_output_mode MUST NOT be used with an alphabet-based scorer.
    • Acoustic models trained with an alphabet file MUST NOT be used with a UTF-8 scorer.

  8. Understand the DeepSpeech RNN architecture

    master

    DeepSpeech uses a Recurrent Neural Network (RNN) to convert audio spectrograms into text transcriptions. The model is designed to run on non-server-class hardware and uses MFCC (Mel-frequency cepstral coefficients) as input features.

    Model Structure:

    • Input: Time-series of MFCC audio features.
    • Layers 1-3: Non-recurrent layers. The first layer uses a context of $C=9$ frames on each side of the current time step. Subsequent non-recurrent layers operate on independent data for each time step.
    • Layer 4: A recurrent layer with forward recurrence ($h^{(f)}$), which must be computed sequentially from the start of the utterance to the end.
    • Layer 5: A non-recurrent layer that takes the forward units as input.
    • Output Layer: Produces logits corresponding to character probabilities for each time slice. For English, the alphabet includes {a, b, c, ..., z, space, apostrophe, blank}.

    Key Technical Details:

    • Activation Function: A clipped Rectified Linear Unit (ReLu): $g(z) = \min{\max{0, z}, 20}$.
    • Loss Function: Connectionist Temporal Classification (CTC) loss, which utilizes the blank character to handle transitions between characters.
    • Training Optimizer: Adam method.

  9. Use Hot-word boosting to influence transcription probability

    master

    DeepSpeech 0.9+ provides a Hot-word boosting API that allows you to increase or decrease the probability of specific words appearing in the transcription. This feature is available in the Model class across all bindings (C, Python, JS, Java, and .Net).

    Core API Methods

    • AddHotWord(word, boost): Adds a word with a specific boost value.
    • EraseHotWord(word): Removes the boost for a specific word.
    • ClearHotWords(): Clears all currently boosted hot-words.

    Usage Constraints and Best Practices

    • Phonetic Relevance: Boosting words that do not exist in the scorer (e.g., many proper nouns) or words that share no phonetic prefix with the input audio will not change the transcription.
    • No Spaces: Hot-words cannot contain spaces. To boost a phrase, you must add each word in the phrase separately.
    • Error Handling: Adding a boost to a word that is already boosted, or attempting to erase a word that was never boosted, will result in an error.
    • Optimal Values: While values vary by use case, it is recommended to keep positive boost values below 20.0. Overly high positive values (e.g., 250.0) may cause the word following the boosted word to be split into individual letters.
    ds = Model(args.model)
# ...
ds.addHotWord(word, boosting)
# ...
print(ds.stt(audio))

  11. Understand kenlm querying data structures

    master

    kenlm supports two primary data structures for querying, both of which use log base 10 for probabilities:

    1. Probing: A probing hash table where keys are 64-bit hashes of n-grams and values are floats. It is the fastest implementation but consumes the most memory.
    2. Trie: A standard trie with bit-level packing to minimize the bits used for word indices and pointers. It uses the least memory and is slightly slower than probing.

    Binary Format and mmap

    To improve performance, you can use a binary format via mmap. Use the ./build_binary script to generate a binary file, then pass the filename to the appropriate Model constructor in your application.

  12. Configure input feature dimensions (n_input)

    master

    The n_input constant defines the number of MFCC features per time-slice of the speech sample. This value should be chosen based on the sample rate of your dataset:

    • 8kHz sample rate: Use 13 features.
    • 16kHz sample rate: Use 26 features (default).

    Adjust n_input to match the dimensionality of the MFCC vectors extracted from your audio data.