DOCSEARCHbyAlgolia
Submit Docs

Montreal Forced Aligner (MFA) Documentation

repository·main·Indexed 23 days ago

https://github.com/montrealcorpustools/montreal-forced-aligner

A command-line utility for performing forced alignment of speech datasets using Kaldi. MFA supports pretrained models, G2P pronunciation generation, and integration with SpeechBrain and WhisperX for transcription, segmentation, and diarization. It provides tools for acoustic model training, dictionary remapping, and alignment analysis with exports available in CSV, TextGrid, and JSON formats.

Tokens
37.8K
Snippets
81
Records
269
Agent score
84%

What's inside Montreal Forced Aligner

  1. What is forced alignment and how does MFA work?

    main

    Forced alignment is a technique that takes an orthographic transcription of an audio file and generates a time-aligned version. It uses a pronunciation dictionary to look up phones (phonemes) for words.

    Montreal Forced Aligner (MFA) performs this process through a four-stage training pipeline:

    1. Monophone models: The first pass models each phone identically, regardless of context.
    2. Triphone models: The second pass accounts for the acoustic context on either side of a phone.
    3. LDA+MLLT: The third pass learns a feature transform to maximize the difference between phone features.
    4. Speaker Adaptation: The final pass enhances the triphone model by accounting for individual speaker differences and calculating a transformation of the Mel Frequency Cepstrum Coefficients (MFCC) features for each speaker.

    MFA is built on the Kaldi ASR toolkit and uses pynini for grapheme-to-phoneme capabilities.

  2. Perform transcription and model training

    main

    MFA 2.0+ supports several core workflows:

    • Transcription: Transcribe a corpus of sound files using an acoustic model, dictionary, and language model.
    • Language Model Training: Train language models from corpora that contain text transcriptions.
    • Dictionary Training: Train pronunciation probability dictionaries from alignments to be used in subsequent alignment or transcription tasks.

  3. Evaluate transcription accuracy in Evaluation mode

    main

    To compare transcriptions against gold-standard references, transcribe a corpus using the same format as for alignment (where each sound file has a corresponding TextGrid or .lab file).

    MFA will transcribe the audio and then align the resulting transcripts with the gold transcriptions using the Bio.pairwise2 alignment algorithm. It calculates the Word Error Rate (WER) and Character Error Rate (CER) for each utterance using the following formula:

    Error rate = (insertions + deletions + (2 * substitutions)) / length_ref

  4. Use Corpus classes for managing data

    main

    Montreal Forced Aligner provides specialized corpus classes to manage different types of data used during alignment and feature extraction:

    • AcousticCorpus: Manages acoustic data (sound files).
    • TextCorpus: Manages text-based data (transcriptions).
    • FileData: A utility class for representing pairs of sound files and their corresponding transcription files.
    • UtteranceData: A utility class for collecting and organizing information about specific utterances within a corpus.
    • AlignmentRemapper: Used to remap existing aligned TextGrid files to a different phone set.

  5. Align using pretrained models

    main

    MFA supports aligning audio using pretrained models. Note the following requirements and behaviors established in version 1.0.0:

    • Dictionary Requirement: Acoustic models no longer contain the dictionary they were trained with. You must specify a dictionary when aligning using pretrained models.
    • Speaker Adaptation: In version 0.8.0, a flag was added to turn off speaker adaptation when aligning with a pretrained model.

  6. How pronunciation probabilities are estimated

    main

    MFA estimates pronunciation probabilities based on the counts of specific pronunciations, normalized by the count of the most frequent pronunciation for that word. This uses add-one smoothing.

    Key details:

    • Normalization: Max normalization is used (dividing by the count of the most frequent pronunciation) to avoid penalizing words that have many different pronunciations.
    • Usage: While these probabilities do not sum to 1, the log of the probabilities is used as summed costs in the lexicon FST, so the lack of summation to 1 is not an issue for the model.
    • Unseen Words: If a word was not present in the training data, pronunciation probabilities are not estimated for its pronunciations.

  9. Extend Corpus functionality with Mixins

    main

    MFA uses a Mixin pattern to provide modular functionality to corpus objects. You can use these mixins to add specific capabilities to your corpus implementations:

    • Base Mixins: CorpusMixin.
    • Acoustic Mixins: AcousticCorpusMixin and AcousticCorpusPronunciationMixin.
    • Text Mixins: TextCorpusMixin and DictionaryTextCorpusMixin.
    • Ivector Mixins: IvectorCorpusMixin.