StyleTTS 2 Documentation

When using the pre-trained models, you must adhere to the following ethical usage agreement:

Requirement: You must inform listeners that the speech samples are synthesized by pre-trained models, unless you have explicit permission to use the voice you are synthesizing.

Specifics:

• You must only use voices whose speakers have granted permission (directly or via license) before making synthesized voices public.
• If you do not have permission, you must publicly announce that the voices are synthesized.

You can import StyleTTS 2 for custom implementations, but note that the core inference depends on a GPL-licensed package which is not included in this repository. Depending on your licensing needs, you have two main options:

1. GPL-licensed approach: Use the GPL-licensed fork which includes an importable script and an experimental streaming API.
2. MIT-licensed approach: Use the styletts2 PyPI package, which uses gruut for phonemization. Note that quality may be lower due to a mismatch between the phonemizer and gruut.

LJSpeech Dataset

• Download and extract the LJSpeech dataset.
• Unzip into the data folder.
• Upsample the audio to 24 kHz (the text aligner and pitch extractor are pre-trained on 24 kHz).

LibriTTS Dataset

• Combine train-clean-360 with train-clean-100.
• Rename the combined folder to train-clean-460.
• Refer to val_list_libritts.txt for formatting examples.

Data List Format

Your data list must follow the format: filename.wav|transcription|speaker. Speaker labels are required for multi-speaker models to sample reference audio for style diffusion training.

Training is performed in two stages. You can run them consecutively.

1. First Stage Training:

   accelerate launch train_first.py --config_path ./Configs/config.yml

2. Second Stage Training (Note: DDP is currently not working; this uses DP):

   python train_second.py --config_path ./Configs/config.yml

Outputs:

• Models are saved as epoch_1st_%05d.pth and epoch_2nd_%05d.pth.
• Checkpoints and Tensorboard logs are saved in the log_dir specified in your config.

To perform single-speaker inference using the LJSpeech corpus, use the provided Jupyter notebook. You will need the pretrained StyleTTS 2 model trained on LJSpeech (24 kHz).

Pre-trained Model Download: https://huggingface.co/yl4579/StyleTTS2-LJSpeech/tree/main

Quick Start: You can run the demo directly in Google Colab via the link provided in the notebook.

https://colab.research.google.com/github/yl4579/StyleTTS2/blob/main/Colab/StyleTTS2_Demo_LJSpeech.ipynb

To set up StyleTTS 2, ensure you have Python >= 3.7 installed. Follow these steps to clone the repository and install dependencies:

1. Clone the repository:

   git clone https://github.com/yl4579/StyleTTS2.git
   cd StyleTTS2

2. Install Python requirements:

   pip install -r requirements.txt

3. Windows Users: Additionally install PyTorch with CUDA support:

   pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 -U

4. For Demo/Phonemization: Install phonemizer and espeak-ng:

   pip install phonemizer
   sudo apt-get install espeak-ng

To perform multi-speaker inference using the LibriTTS corpus, follow these steps:

1. Download Pre-trained Model: Get the LibriTTS pretrained model from https://huggingface.co/yl4579/StyleTTS2-LibriTTS/tree/main.
2. Download Reference Audio: Download reference_audio.zip from https://huggingface.co/yl4579/StyleTTS2-LibriTTS/resolve/main/reference_audio.zip.
3. Setup Directory: Unzip reference_audio.zip into the demo directory before running the inference notebook.
4. Run Notebook: Use the Inference_LibriTTS.ipynb notebook for the execution logic.

https://colab.research.google.com/github/yl4579/StyleTTS2/blob/main/Colab/StyleTTS2_Demo_LibriTTS.ipynb

Use the following script to finetune using a pre-trained LibriTTS checkpoint. Ensure the checkpoint is downloaded and unzipped in the appropriate folder.

Standard Finetuning (DP)

python train_finetune.py --config_path ./Configs/config_ft.yml

Single GPU Finetuning (Accelerate)

If using a single GPU, use this version to save VRAM and increase speed:

accelerate launch --mixed_precision=fp16 --num_processes=1 train_finetune_accelerate.py --config_path ./Configs/config_ft.yml

Note on OOM: If you encounter Out of Memory errors after joint_epoch, you can skip SLM adversarial training by setting joint_epoch to a value larger than epochs.

When editing Configs/config.yml, pay attention to these key parameters:

Before starting finetuning, you can modify the Configs/config_ft.yml file. Key parameters to adjust based on your GPU memory (VRAM) include batch_size, max_len, and loss_params['joint_epoch']. For example, to avoid Out-of-Memory (OOM) errors on limited hardware, reduce the batch size and max length.

config_path = "Configs/config_ft.yml"

import yaml
config = yaml.safe_load(open(config_path))

config['data_params']['root_path'] = "Data/wavs"
config['batch_size'] = 2 # reduce if low RAM
config['max_len'] = 100 # reduce if low RAM
config['loss_params']['joint_epoch'] = 110 # skip SLM adversarial training if low RAM

with open(config_path, 'w') as outfile:
  yaml.dump(config, outfile, default_flow_style=True)