DOCSEARCHbyAlgolia
Submit Docs

Charsiu Phonetic Aligner

repository·main·Indexed 43 days ago

https://github.com/lingjzhu/charsiu

A transformer-based phonetic aligner for textless alignment (recognizing phonemes from audio) and forced alignment (aligning text to audio). It features multiple aligner types including charsiu_attention_aligner, charsiu_chain_attention_aligner, charsiu_chain_forced_aligner, and charsiu_predictive_aligner. Supports English and Mandarin Chinese with the ability to export results to TextGrid files via the .serve() method.

Tokens
10.8K
Snippets
43
Records
47
Agent score
56%

What's inside Charsiu

  1. Access English Common Voice alignments and audio

    main

    The Charsiu project provides phone- and word-level alignments (TextGrids) for the English subset of the Common Voice dataset (~2000 hours).

    • Alignments: Available as TextGrids in a Google Drive folder. These were generated using charsiu_forced_aligner with the model charsiu/en_w2v2_fc_10ms.
    • Audio: Can be downloaded from the Common Voice Project or accessed via the HuggingFace Hub.

    Note: Loading the dataset from HuggingFace requires approximately 80GB of memory and authentication.

    from datasets import load_dataset

# Load English Common Voice 8.0 from HuggingFace
dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", split='train', use_auth_token=True)

  2. Access Mandarin Chinese speech alignments and audio

    main

    Charsiu provides phone- and word-level alignments for several Mandarin Chinese datasets.

  3. Audio requirements and best practices

    main

    To ensure optimal performance and avoid crashes with Charsiu, follow these audio processing guidelines:

    1. Sampling Rate: While there is no strict requirement, it is recommended to resample audio to 16,000 Hz (16k) for compatibility with many models.
    2. Audio Duration: Charsiu works best with short audio segments.
      • Ideal: Around 10 ms.
      • Warning: If files are $\ge$ 15 ms, the program may crash.
    3. File Format: Use .wav files.

  4. Initialize a Charsiu model for textless alignment

    main

    You can perform textless alignment by importing a pre-trained model from the charsiu library and initializing it. This allows the model to align audio to phonetic representations without requiring explicit text input during the alignment process.

    Step 1: Import and initialize

    import charsiu

# Example: Initialize a selected model
# Note: Replace 'model_name' with the specific model identifier you wish to use
model = charsiu.load_model('model_name')
    import charsiu

# Initialize selected model from Charsiu
model = charsiu.load_model('model_name')

  5. Install Charsiu dependencies and library

    main

    To use Charsiu, you must install the core machine learning libraries and specific phonetic/audio processing tools. If you are working in a notebook environment like Colab, you also need to clone the repository from the development branch.

    1. Install Python dependencies:

      • torch, torchvision, torchaudio
      • datasets, transformers
      • g2p_en, praatio, librosa

    2. Clone the repository: git clone -b development https://github.com/lingjzhu/charsiu

    !pip install torch torchvision torchaudio
!pip install datasets transformers
!pip install g2p_en praatio librosa

!git clone -b development https://github.com/lingjzhu/charsiu

  6. Perform textless alignment with Charsiu

    main

    Textless alignment allows you to align an audio file to time segments without requiring a transcript.

    To use this feature:

    1. Import charsiu_predictive_aligner from Charsiu.
    2. Initialize the model by specifying an aligner (e.g., 'charsiu/en_w2v2_fc_10ms').
    3. Use .align(audio=...) to get the alignment results in memory.
    4. Alternatively, use .serve(audio=..., save_to=...) to perform alignment and save the results directly to a TextGrid file.

    Important: You must ensure the charsiu package directory is in your sys.path and that you provide absolute paths to your audio and output files.

    import os
import sys
from Charsiu import charsiu_predictive_aligner

# Setup path to charsiu package
charsiu_dir = 'path/to/charsiu/'
os.chdir(charsiu_dir)
sys.path.append('%s/src/' % charsiu_dir)

# Initialize model
charsiu = charsiu_predictive_aligner(aligner='charsiu/en_w2v2_fc_10ms')

# Define file paths
audio_file = 'path/to/your_audio.WAV'
textgrid_file = 'path/to/output.TextGrid'

# Option 1: Get alignment in memory
alignment = charsiu.align(audio=audio_file)
print(alignment)  # Prints (start_time, end_time, 'segment_label')

# Option 2: Save directly to a TextGrid file
charsiu.serve(audio=audio_file, save_to=textgrid_file)

  7. Install Charsiu and essential libraries on Colab

    main

    To use Charsiu in a Google Colab environment, you must first install the necessary system and Python dependencies, then download the Charsiu library itself.

    Step 1: Install essential libraries

    Run the following command to install the required base libraries:

    # (Command from notebook segment)

    Step 2: Download Charsiu library

    Download the repository to your environment:

    # (Command from notebook segment)

    Step 3: Import libraries

    Once installed, import the necessary modules to begin working with the models.

    # Note: The specific shell commands are encoded in the provided notebook segment.
# Based on the headings, the workflow is:
# 1. !pip install <essential_libraries>
# 2. !git clone <charsiu_repo>
# 3. import <modules>

  8. Install Charsiu and dependencies

    main

    To use Charsiu, you must install its core dependencies. If running on Colab, you should also clone the repository and add the src directory to your system path.

    Core Dependencies:

    • torch, torchvision, torchaudio
    • datasets, transformers
    • praatio (Note: version 5.x is required for TextGrid functions)
    • librosa
    • g2p_en, g2pM

    Colab Setup Steps:

    1. Install libraries via pip.
    2. Clone the Charsiu repository from GitHub.
    3. Add the src directory to sys.path to allow importing from the Charsiu package.
    # Step 1: install essential libraries
!pip install torch torchvision torchaudio
!pip install datasets transformers
!pip install praatio librosa
!pip install g2p_en g2pM

# Step 2: download Charsiu library
import os
from os.path import exists, join, expanduser

os.chdir(expanduser("~"))
charsiu_dir = 'charsiu'
if exists(charsiu_dir):
  !rm -rf /root/charsiu
if not exists(charsiu_dir):
  ! git clone -b development https://github.com/lingjzhu/$charsiu_dir
  ! cd charsiu && git checkout && cd -

os.chdir(charsiu_dir)

# Step 3: import essential libraries
import sys
import torch
from itertools import groupby

sys.path.insert(0,'src')

  9. Install Charsiu and essential libraries in Colab

    main

    To use Charsiu in a Google Colab environment, you must first install the necessary system and Python dependencies, then download the Charsiu library itself.

    Step 1: Install essential libraries

    !pip install torch torchaudio

    Step 2: Download Charsiu library

    !pip install git+https://github.com/lingjzhu/charsiu.git

    Step 3: Import libraries

    import torch
import torchaudio
import charsiu
    !pip install torch torchaudio
!pip install git+https://github.com/lingjzhu/charsiu.git

  10. Install and setup Charsiu on Colab

    main

    To run Charsiu in a Google Colab environment, follow these steps:

    1. Install essential libraries: Run the cell to install the required dependencies.
    2. Download Charsiu library: Clone or download the Charsiu repository to your environment.
    3. Import libraries: Import the necessary Python modules to begin using the aligner.

    Note: You can also mount your Google Drive to use your own datasets.

  11. Install Charsiu via Colab

    main

    To use Charsiu in a Google Colab environment, follow these steps to install the necessary dependencies and the library itself:

    1. Install essential libraries: Install the base requirements for the environment.
    2. Download Charsiu: Clone or download the Charsiu repository.
    3. Import libraries: Import the required Python packages to begin using the model.

    [optional] You can also mount your Google Drive to use your own datasets.

    # Step 1: install essential libraries
!pip install torch torchaudio

# Step 2: download Charsiu library
!git clone https://github.com/lingjzhu/charsiu.git

# Step 3: import essential libraries
import torch
import torchaudio
import sys
sys.path.append("./charsiu")
import charsiu

  12. Initialize the Charsiu predictive aligner

    main

    The charsiu_predictive_aligner function initializes the alignment model. It requires an aligner parameter which specifies the pre-trained model to use.

    Example model string: 'charsiu/en_w2v2_fc_10ms'

    from Charsiu import charsiu_predictive_aligner

charsiu = charsiu_predictive_aligner(aligner='charsiu/en_w2v2_fc_10ms')