SpeechT5 Documentation

SpeechT5 is a unified-modal framework based on the encoder-decoder architecture, inspired by the T5 (Text-To-Text Transfer Transformer) model. It is designed for self-supervised speech and text representation learning.

Core Components:

1. Pre-nets: Six modal-specific (speech or text) pre-nets that preprocess the input.
2. Shared Encoder-Decoder: A central network that models the sequence-to-sequence transformation.
3. Post-nets: Modal-specific post-nets that generate the final output in the target modality (speech or text) based on the decoder's output.

Cross-Modal Alignment: To align speech and text into a unified semantic space, SpeechT5 uses a cross-modal vector quantization approach. This method randomly mixes speech/text states with latent units to serve as the interface between the encoder and the decoder.

To set up the SpeechT5 environment, you must initialize the fairseq submodule, install it in editable mode, and install espnet.

git submodule update --init SpeechT5/fairseq
cd SpeechT5/
pip install --editable fairseq/
pip install espnet

To prepare the environment for fine-tuning or pre-training, initialize the fairseq submodule and install it in editable mode, then install sacrebleu.

git submodule update --init SpeechLM/fairseq
cd SpeechLM/
pip install --editable fairseq/
pip install sacrebleu==1.5.1

To perform inference, copy the wavllm directory into the fairseq/examples directory and execute the inference_sft.sh script.

Command Syntax: bash examples/wavllm/scripts/inference_sft.sh $model_path $data_name

Replace $model_path with the path to your downloaded model and $data_name with the name of the task/data you wish to run.

cp -r wavllm fairseq/examples
cd fairseq
bash examples/wavllm/scripts/inference_sft.sh $model_path $data_name

Fine-tuning can be performed on downstream tasks like AVSR. For example, to fine-tune an AVSR model using 30 hours of labeled data:

cd VATLM/vat_hubert/vathubert/scripts/finetune_avsr
ngpu=8
updatefreq=1
save_path=/path/to/save_path

bash base_lrs3_finetune30_av.sh ${ngpu} ${updatefreq} ${save_path}

A cascaded Speech Translation system can be built by fine-tuning an ASR model and an MT model separately.

1. ASR Model

Fine-tune from the combination of Hubert Large and mBART model. Important: You must update mbart_path and hubert_path in the .sh file to your local paths.

bash YiTrans/exp_scripts/finetune_ASR/finetune_hubert24_mbart24_en.sh

2. MT Model

Fine-tune from the mBART model. Important: You must update mbart_path in the .sh file to your local path.

bash YiTrans/exp_scripts/finetune_MT/finetune_mbart_en-de.sh

Data preparation depends on the modality being used:

1. Audio or Visual data: Follow the AV-HuBERT preparation scripts to pre-process data and generate train.tsv and train.km files.
2. Unimodal Audio data: Replace the visual modality with a zero vector. Extract features using the AV-HuBERT preparation script and perform k-means clustering to obtain labels.
3. Unimodal Text data:
   • Use a small amount of paired text-audio data to obtain paired phone-unit data.
   • Obtain phoneme sequences via the provided lexicon.
   • Extract features and perform k-means clustering to get unit data.
   • Train the phone2unit model using the SpeechLM hidden unit tokenizer script.

After fine-tuning, use the inference scripts to decode the models. You can specify a generation set (e.g., dev) and a beam_size.

• For the Base model, use speechlm/scripts/tune_speechlm_st/inference_base.sh.
• For the Large model, use speechlm/scripts/tune_speechlm_st/inference_large.sh.

# Decode Base model
model_path=path/to/your/fine-tuned/model
lang=de
data_dir=dataset/CommonVoice/v4/en/en-${lang}
bash speechlm/scripts/tune_speechlm_st/inference_base.sh $model_path $data_dir $lang dev

# Decode Large model
bash speechlm/scripts/tune_speechlm_st/inference_large.sh $model_path $data_dir $lang dev

Data preparation for SpeechT5 requires two main components:

1. Speech Manifests: Follow the wav2vec 2.0 manifest preparation steps.
2. HuBERT Labels: Prepare HuBERT labels using K-means clustering.

SpeechT5 requires a third column in the manifest for speaker embeddings. Pre-computed speaker embeddings for the 960hr LibriSpeech training and dev-other data are available via the project's provided links. Example manifests are also provided in the repository's documentation.