DOCSEARCHbyAlgolia
Submit Docs

pyctcdecode

repository·main·Indexed 19 days ago

https://github.com/kensho-technologies/pyctcdecode

A fast, feature-rich CTC beam search decoder for speech recognition written in Python. It supports n-gram (KenLM) language models, BPE vocabulary, hotword boosting, and real-time decoding. The library provides tools for shallow fusion, batch decoding via multiprocessing, and integration with Nvidia NeMo Conformer-CTC models.

Tokens
9.3K
Snippets
34
Records
42
Agent score
66%

What's inside pyctcdecode

  2. Quick Start: Build and use a CTC decoder

    main

    To perform CTC beam search decoding with shallow fusion, use build_ctcdecoder to initialize a decoder with your alphabet labels and a KenLM language model. You can then call .decode(logits) to get the transcript.

    Note that pyctcdecode automatically handles BPE (Byte Pair Encoding) token merging if the provided labels are BPE-based, though the language model itself remains word-based.

    from pyctcdecode import build_ctcdecoder

# specify alphabet labels as they appear in logits
labels = [
    " ", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l",
    "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z",
]

# prepare decoder and decode logits via shallow fusion
decoder = build_ctcdecoder(
    labels,
    kenlm_model_path="/my/dir/kenlm_model.arpa",  # either .arpa or .bin file
    alpha=0.5,  # tuned on a val set
    beta=1.0,  # tuned on a val set
)
text = decoder.decode(logits)

  3. Compare greedy decoding vs pyctcdecode

    main

    A simple greedy CTC decode involves taking the argmax of the logits at each time step and squashing repeated characters. pyctcdecode provides a more advanced implementation via build_ctcdecoder.

    # Manual greedy decoding implementation for comparison
def _greedy_decode(logits, labels):
    """Decode argmax of logits and squash in CTC fashion."""
    label_dict = {n: c for n, c in enumerate(labels)}
    prev_c = None
    out = []
    for n in logits.argmax(axis=1):
        c = label_dict.get(n, "")  # if not in labels, then assume it's ctc blank char
        if c != prev_c:
            out.append(c)
        prev_c = c
    return "".join(out)

# Using pyctcdecode instead
from pyctcdecode import build_ctcdecoder
decoder = build_ctcdecoder(labels)
result = decoder.decode(logits)

  4. How BeamSearchDecoderCTC serialization works

    main

    The BeamSearchDecoderCTC class uses a directory-based structure for persistence. When using save_to_dir, the following components are stored:

    1. Alphabet: Serialized using Alphabet.dumps() and saved to a file defined by _ALPHABET_SERIALIZED_FILENAME.
    2. Language Model: If present, the language model is saved to a directory defined by _LANGUAGE_MODEL_SERIALIZED_DIRECTORY using its own save_to_dir method.

    When loading via load_from_dir, the class uses parse_directory_contents to validate that the required alphabet file and optional language model directory exist before reconstruction.

  5. How LanguageModel scoring works

    main

    The LanguageModel.score method performs the following steps:

    1. Calculates the base KenLM score for the word given the prev_state.
    2. If the word is not in the provided unigrams set or the KenLM model, it applies the unk_score_offset.
    3. If is_last_word is True and score_boundary is enabled, it adds the end-of-sentence boundary score.
    4. Applies the shallow fusion formula: score = alpha * lm_score * LOG_BASE_CHANGE_FACTOR + beta.

  8. Integrate pyctcdecode with Hugging Face Transformers

    main

    To use pyctcdecode with a Hugging Face model (like Wav2Vec2), you must first extract the logits and the vocabulary from the model. The workflow involves:

    1. Loading a pretrained model and processor using transformers.
    2. Processing an audio signal to obtain logits.
    3. Extracting the vocabulary from the processor's tokenizer.
    4. Passing the vocabulary and logits to pyctcdecode.

    Note: pyctcdecode automatically attempts to handle unconventional vocabulary shapes (such as specific CTC blank tokens like 氻), but it may issue a warning if it is uncertain about the mapping.

    import soundfile as sf
from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC
from pyctcdecode import build_ctcdecoder

# 1. Load model and processor
asr_processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-base-960h")
asr_model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")

# 2. Prepare audio and get logits
arr, _ = sf.read('audio_file.wav')
input_values = asr_processor(arr, return_tensors="pt", sampling_rate=16000).input_values
logits = asr_model(input_values).logits.cpu().detach().numpy()[0]

# 3. Get vocabulary
vocab_list = list(asr_processor.tokenizer.get_vocab().values())

# 4. Decode using pyctcdecode
decoder = build_ctcdecoder(vocab_list)
result = decoder.decode(logits)
print(result)

  9. Integrate pyctcdecode with NeMo models

    main

    To use pyctcdecode with an NVIDIA NeMo ASR model, you must extract the logits from the model and use the model's vocabulary to build the decoder.

    1. Install NeMo: Ensure you have the NeMo toolkit installed.
    2. Load Model: Load a pretrained NeMo model (e.g., EncDecCTCModel).
    3. Extract Logits: Use asr_model.transcribe(..., logprobs=True) to obtain the logit matrix.
    4. Build Decoder: Pass asr_model.decoder.vocabulary to build_ctcdecoder.
    5. Decode: Call .decode(logits) on the resulting decoder instance.
    import nemo.collections.asr as nemo_asr
from pyctcdecode import build_ctcdecoder

# Load model
asr_model = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name='QuartzNet15x5Base-En')

# Transcribe to logits
logits = asr_model.transcribe(["audio_file.wav"], logprobs=True)[0]

# Build and use decoder
decoder = build_ctcdecoder(asr_model.decoder.vocabulary)
text = decoder.decode(logits)

  11. Compare pyctcdecode performance against DeepSpeech decoder

    main

    The tutorial demonstrates how to benchmark pyctcdecode against the ds_ctcdecoder (DeepSpeech) implementation. This involves sweeping through different beam_width values and measuring both Word Error Rate (WER) and average runtime per sample (ms).

    # Example logic for sweeping beam_width in pyctcdecode
for beam_width in [1, 5, 10, 50, 100, 150, 200]:
    decoder = build_ctcdecoder(
        labels,
        kenlm_model,
        unigrams,
        alpha=0.7,
        beta=3.0,
        score_lm_boundary=True,
    )
    with multiprocessing.get_context("fork").Pool(15) as pool:
        pred_list = decoder.decode_batch(pool, logits_list, beam_width=beam_width)
    # Calculate WER and timing...

  12. Perform batch decoding using multiprocessing

    main

    To decode a list of logits efficiently, use the .decode_batch() method by providing a multiprocessing pool.

    import multiprocessing

with multiprocessing.get_context("fork").Pool() as pool:
    text_list = decoder.decode_batch(pool, logits_list)