DOCSEARCHbyAlgolia
Submit Docs

F5-TTS-Thai: Thai Text-to-Speech

repository·main·Indexed 41 days ago

https://github.com/vyncx/f5-tts-thai

An open-source Thai language Text-to-Speech (TTS) tool based on the F5-TTS Flow Matching model. It features two model versions: v1 for natural Thai prosody and v2 for improved reading accuracy via IPA. The package includes a Gradio WebUI, a CLI for inference, a socket server for real-time streaming, and tools for finetuning and training using architectures like UNet, DiT, and MMDit.

Tokens
14K
Snippets
43
Records
64
Agent score
51%

What's inside F5-TTS-Thai

  1. Overview of F5-TTS Backbone Architectures

    main

    The F5-TTS project utilizes three primary backbone architectures for its model implementations: unett.py, dit.py, and mmdit.py. These architectures differ in their structural approach to handling text embeddings, noise, and conditioning.

    • unett.py: A flat UNet transformer. It follows the structure used in the E2-TTS and Voicebox papers but utilizes rotary positional embeddings. It supports absolute positional embeddings and ConvNextV2 blocks for embedded text prior to concatenation.
    • dit.py: An AdaLN-Zero Diffusion Transformer (DiT). It uses embedded timesteps as conditioning. The input is a concatenation of noised_input, masked_cond, and embedded_text, followed by a linear projection. It supports absolute positional embeddings, ConvNextV2 blocks for embedded text, and optional long skip connections from the first layer to the last layer.
    • mmdit.py: A structure based on Stable Diffusion 3 (SD3). It uses timesteps as conditioning and employs a dual-stream approach: a left stream for text embeddings (applying absolute positional embeddings) and a right stream for the concatenation of masked_cond and noised_input (using the same convolutional positional embeddings as unett).

  2. Best practices for Thai TTS generation

    main

    To improve the quality and accuracy of the generated Thai speech, follow these recommendations:

    • Reference Audio: Use a reference audio clip that is no longer than 8 seconds and ideally free of background noise.
    • Reference Text: For ref_text or non-Thai words, use Thai phonetic transcriptions (e.g., use "กู้ดมอร์นิ่ง" instead of "Good Morning") to improve pronunciation.
    • Sentence Structure: Use spaces between sentences to allow the model to segment the generation process.
    • Accuracy vs. Speed:
      • To reduce reading errors, you can set a max_chars limit (maximum characters per segment), though this slows down generation.
      • You can decrease the step (NFE) value to increase speed.
    • Speed Adjustment: Reducing the speed parameter (e.g., to 0.8 or 0.9) can help prevent word skipping or truncated words, but setting it too low may cause the reference audio characteristics to bleed into the output.

  3. Best practices for high-quality TTS generation

    main

    To ensure optimal audio quality and avoid inference failures, follow these guidelines:

    • Reference Audio: Use audio shorter than 15s. Include approximately 1s of silence at the end of the reference audio to prevent the model from truncating words mid-speech.
    • Casing: Use lowercase letters for normal words. Uppercased letters will be uttered letter-by-letter (e.g., 'ABC' becomes 'A-B-C').
    • Pauses: Explicitly introduce pauses using spaces (" ") or punctuation (, or .).
    • Numbers: Preprocess numbers into words (e.g., convert '123' to 'one hundred twenty three') to ensure correct pronunciation.
    • Troubleshooting Silence: If the output is pure silence, verify that ffmpeg is installed on your system.
    • Finetuned Models: If using an early-stage finetuned checkpoint, try setting use_ema to false.

  4. Use the Socket Realtime Client

    main

    F5-TTS supports real-time streaming via a socket server.

    1. Start the server: Run python src/f5_tts/socket_server.py.
    2. Run the client: Use a socket client to send text to the server and receive an audio stream. The client must handle the incoming byte stream as float32 audio data and play it using a library like pyaudio.

    Prerequisites for Client: If using Linux, you may need to install portaudio development headers:

    sudo apt-get install portaudio19-dev
pip install pyaudio
    # Example client snippet for listening to the stream
import socket
import asyncio
import pyaudio
import numpy as np

async def listen_to_F5TTS(text, server_ip="localhost", server_port=9998):
    # ... connection and stream handling logic ...
    # Receives bytes, converts to np.float32, and writes to pyaudio stream

  5. Download pretrained model checkpoints

    main

    The project uses pretrained model checkpoints for inference. You can find the base models on Hugging Face at https://huggingface.co/SWivid/F5-TTS.

    When setting up your local environment, ensure you have the following directory structure for the checkpoints:

    ckpts/
    E2TTS_Base/
        model_1200000.pt
    F5TTS_Base/
        model_1200000.pt
    https://huggingface.co/SWivid/F5-TTS

  6. Prepare specific datasets for training

    main

    Before training, you must prepare your datasets using the provided scripts. Ensure you have downloaded the corresponding datasets first and updated the paths within the scripts. The following scripts are available for specific datasets:

    • Emilia: src/f5_tts/train/datasets/prepare_emilia.py
    • Wenetspeech4TTS: src/f5_tts/train/datasets/prepare_wenetspeech4tts.py
    • LibriTTS: src/f5_tts/train/datasets/prepare_libritts.py
    • LJSpeech: src/f5_tts/train/datasets/prepare_ljspeech.py

    If you are building a custom dataset, you can use src/f5_tts/train/datasets/prepare_csv_wavs.py which requires a metadata.csv file.

    # Prepare the Emilia dataset
python src/f5_tts/train/datasets/prepare_emilia.py

# Prepare the Wenetspeech4TTS dataset
python src/f5_tts/train/datasets/prepare_wenetspeech4tts.py

# Prepare the LibriTTS dataset
python src/f5_tts/train/datasets/prepare_libritts.py

# Prepare the LJSpeech dataset
python src/f5_tts/train/datasets/prepare_ljspeech.py

# Create custom dataset with metadata.csv
python src/f5_tts/train/datasets/prepare_csv_wavs.py

  7. Finetune F5-TTS

    main

    Finetuning can be performed via command line or through a Gradio UI.

    • Gradio UI: Use src/f5_tts/train/finetune_gradio.py for a visual training/finetuning interface.
    • Best Practice: When performing early-stage finetuning (only a few updates), it is recommended to set use_ema = False. Keeping use_ema = True can be harmful because the Exponential Moving Average weights are still dominated by the pretrained model, preventing the checkpoint from adapting quickly to the new data.

  8. Install f5-tts-th

    main

    You can install the package directly via pip or by cloning the repository. For optimal GPU performance, it is highly recommended to install specific versions of torch and torchaudio with CUDA 11.8 support.

    Prerequisites:

    • Python 3.10 or higher
    • CUDA 11.8 (recommended)

    Option 1: Direct Pip Install

    pip install f5-tts-th

    Option 2: Clone and Install from Source

    git clone https://github.com/VYNCX/F5-TTS-THAI.git
cd F5-TTS-THAI
python -m venv venv
# On Windows:
call venv/scripts/activate
# On Linux/macOS:
source venv/bin/activate

pip install git+https://github.com/VYNCX/F5-TTS-THAI.git

# Install GPU optimized dependencies
pip install torch==2.3.0+cu118 torchaudio==2.3.0+cu118 --extra-index-url https://download.pytorch.org/whl/cu118

    Alternatively, you can run the provided install.bat file on Windows.

    pip install f5-tts-th

  9. Train the F5-TTS pretrained model

    main

    Training uses accelerate to manage distributed training and precision.

    1. Configure Accelerate: Run accelerate config to set up your environment (e.g., multi-GPU DDP, fp16). This saves the configuration to ~/.cache/huggingface/accelerate/default_config.yaml.
    2. Launch Training: Use accelerate launch pointing to the training script and a configuration file from src/f5_tts/configs.

    You can override Hydra configuration parameters using the ++ prefix (e.g., ++datasets.batch_size_per_gpu=19200) or pass flags directly to accelerate (e.g., --mixed_precision=fp16).

    # 1. Setup accelerate config
accelerate config

# 2. Launch training with a config file
accelerate launch src/f5_tts/train/train.py --config-name F5TTS_Base_train.yaml

# 3. Launch training with overrides (mixed precision and batch size)
accelerate launch --mixed_precision=fp16 src/f5_tts/train/train.py --config-name F5TTS_Small_train.yaml ++datasets.batch_size_per_gpu=19200

  10. Train and Finetune the model

    main

    To finetune the model, you can use the provided Google Colab notebook or install the package in editable mode and launch the Gradio training interface.

    Local Setup for Finetuning:

    git clone https://github.com/VYNCX/F5-TTS-THAI.git
cd F5-TTS-THAI
pip install -e .

# Launch the training interface
f5-tts_finetune-gradio
    f5-tts_finetune-gradio

  11. Contribute new pretrained or finetuned models

    main

    If you have developed a new pretrained or finetuned model, you can contribute it to the community by following these steps:

    1. Prepare Files: Ensure you have a model checkpoint (it is recommended to use a pruned version containing only ema_model_state_dict) and the corresponding vocab file used for tokenization.
    2. Host on Hugging Face: Upload your model files to a public Hugging Face model repository.
    3. Submit a Pull Request: Add a new model card entry to src/f5_tts/infer/SHARED.md following the existing format.

  12. Launch the Gradio Web Interface

    main

    You can launch a web-based GUI for inference using the f5-tts_infer-gradio command. This interface supports basic TTS with chunk inference, multi-style/multi-speaker generation, and Voice Chat powered by Qwen2.5-3B-Instruct.

    Key flags:

    • --inbrowser: Automatically opens the interface in your default web browser.
    • --root_path "/path": Sets the root path if the application is not served from the domain root (e.g., https://example.com/myapp).
    # Launch with automatic browser opening
f5-tts_infer-gradio --inbrowser

# Launch with a specific root path
f5-tts_infer-gradio --root_path "/myapp"