DOCSEARCHbyAlgolia
Submit Docs

F5-TTS: Text-to-Speech with Flow Matching

repository·main·Indexed 12 days ago

https://github.com/swivid/f5-tts

A high-performance text-to-speech system using Diffusion Transformers with ConvNeXt V2 and Flow Matching for fluent and faithful speech synthesis. Version 1.1.22 supports inference via CLI and Gradio Web UI, as well as training and finetuning. It provides capabilities for multi-style and multi-speaker generation, voice chat powered by Qwen2.5-3B-Instruct, and objective evaluation metrics including WER, SIM, and UTMOS.

Tokens
19.5K
Snippets
61
Records
83
Agent score
97%

What's inside F5-TTS

  1. Best practices for high-quality inference

    main

    To avoid suboptimal generation or failures, follow these guidelines:

    • Reference Audio: Use audio <12s and include a small amount of silence (e.g., 1s) at the end to prevent mid-word truncation.
    • Casing: Use <ins>UPPERCASE</ins> (e.g., K.F.C.) to force letter-by-letter pronunciation. Use lowercase for common words.
    • Pauses: Use spaces (" ") or punctuation (,, .) to explicitly introduce pauses.
    • Sentence Boundaries: Ensure a space follows English punctuation marks at the end of a sentence to ensure correct chunking.
    • Numbers: Preprocess numbers into Chinese characters if you want them read in Chinese; otherwise, they will be read in English.
    • Silence Issues: If the output is pure silence, verify that FFmpeg is installed.
    • Finetuned Models: If using an early-stage finetuned checkpoint, try turning off use_ema.

  2. Understand the available backbone architectures in F5-TTS

    main

    F5-TTS utilizes different backbone architectures for its flow matching process. Depending on your implementation needs, you can choose between three primary structures:

    1. UNet Transformer (unett.py): A flat UNet transformer. It follows the structure used in E2-TTS and the Voicebox paper but utilizes rotary positional embeddings. It supports absolute positional embeddings and ConvNextV2 blocks for embedded text prior to concatenation.

    2. DiT (dit.py): An AdaLN-Zero Diffusion Transformer (DiT). It uses embedded timesteps as conditioning. The input is a concatenation of noised_input, masked_cond, and embedded_text, followed by a linear projection. It supports absolute positional embeddings, ConvNextV2 blocks for embedded text, and optional long skip connections (connecting the first layer to the last layer).

    3. MMDiT (mmdit.py): A structure based on Stable Diffusion 3. It uses timesteps as conditioning and employs a dual-stream approach:

      • Left stream: Processes text embeddings with absolute positional embeddings.
      • Right stream: Processes concatenated masked_cond and noised_input using the same convolutional positional embeddings as the UNet architecture.

  3. Install F5-TTS via Conda and Pip

    main

    To install F5-TTS, first create a dedicated environment with Python 3.10 or higher and install FFmpeg. Then, install PyTorch compatible with your hardware (NVIDIA, AMD, Intel, or Apple Silicon) before installing the F5-TTS package.

    1. Environment Setup

    # Create a conda env with python_version>=3.10
conda create -n f5-tts python=3.11
conda activate f5-tts

# Install FFmpeg
conda install ffmpeg

    2. Install PyTorch

    Select the command corresponding to your device (see full documentation for specific version/index details).

    3. Install F5-TTS

    Choose one of the following methods:

    • For inference only: Install as a pip package.
    • For training/finetuning: Install in editable mode from the source repository.
    # For inference only
pip install f5-tts

# For training/finetuning (Local editable)
git clone https://github.com/SWivid/F5-TTS.git
cd F5-TTS
pip install -e .

  4. Install PyTorch for different hardware

    main

    Depending on your GPU or accelerator, use the appropriate installation command for PyTorch and Torchaudio.

    • NVIDIA GPU: Use the specific CUDA index URL (e.g., cu128 or cu124).
    • AMD GPU (Linux only): Use the ROCm index URL. Note that RDNA 3.5 and RDNA 4 architectures require ROCm 7.x.
    • Intel GPU: Use the XPU index URL. Requires Intel® Deep Learning Essentials or Intel® oneAPI Base Toolkit (or use IPEX).
    • Apple Silicon: Use the standard stable PyTorch installation.
    # NVIDIA GPU example
pip install torch==2.8.0+cu128 torchaudio==2.8.0+cu128 --extra-index-url https://download.pytorch.org/whl/cu128

# AMD GPU example
pip install torch==2.9.1+rocm7.2 torchaudio==2.9.1+rocm7.2 --extra-index-url https://download.pytorch.org/whl/rocm7.2

# Intel GPU example
pip install torch torchaudio --index-url https://download.pytorch.org/whl/test/xpu

# Apple Silicon
pip install torch torchaudio

  5. Prepare specific datasets for training

    main

    F5-TTS provides dedicated preparation scripts for several common datasets. You must download the datasets first and update the paths within the scripts before running them.

    Available scripts:

    • Emilia
    • Wenetspeech4TTS
    • LibriTTS
    • LJSpeech
    # Prepare the Emilia dataset
python src/f5_tts/train/datasets/prepare_emilia.py

# Prepare the Wenetspeech4TTS dataset
python src/f5_tts/train/datasets/prepare_wenetspeech4tts.py

# Prepare the LibriTTS dataset
python src/f5_tts/train/datasets/prepare_libritts.py

# Prepare the LJSpeech dataset
python src/f5_tts/train/datasets/prepare_ljspeech.py

  6. Build TensorRT-LLM Engines and Launch Server

    main

    Once inside the Docker container, you must build the TensorRT-LLM engines for the Qwen and Whisper models. This is done using the run.sh script.

    Model Naming Convention

    Ensure you use the correct model version prefix:

    • Use F5TTS_v1_* for version 1 models (e.g., F5TTS_v1_Base).
    • Use F5TTS_* for version 0 models (e.g., F5TTS_Base).

    Custom Checkpoints

    • To use a custom checkpoint, set the ckpt_file and vocab_file variables within run.sh.
    • Important: If you trained or fine-tuned using fp32, you must add the --dtype float32 flag during the checkpoint conversion phase in run.sh.
    • If your checkpoint has a different structure, you may need to modify scripts/convert_checkpoint.py.
    # Example: Build engines and launch for F5TTS_v1_Base
# Syntax: bash run.sh <arg1> <arg2> <model_name>
bash run.sh 0 4 F5TTS_v1_Base

  7. Run Real-time Socket Service

    main

    F5-TTS supports real-time voice output with chunk streaming via a socket server/client architecture.

    1. Start the server: python src/f5_tts/socket_server.py

    2. Run the client: python src/f5_tts/socket_client.py

    Note: If PyAudio is not installed, you may need to install system dependencies (sudo apt-get install portaudio19-dev) and then pip install pyaudio.

    # Start socket server
python src/f5_tts/socket_server.py

# Communicate with socket client
python src/f5_tts/socket_client.py

  8. Create a custom dataset using a CSV file

    main

    To use your own data, prepare a CSV file with a header containing two columns: audio_file|text. Note that audio_file paths must be absolute. Use the prepare_csv_wavs.py script to process this into the required format.

    python src/f5_tts/train/datasets/prepare_csv_wavs.py /path/to/metadata.csv /path/to/output

  9. Train or Finetune F5-TTS

    main

    F5-TTS supports training and finetuning. You can use Hugging Face accelerate for distributed training or use the Gradio web interface for a quick start.

    Finetuning via Gradio

    f5-tts_finetune-gradio

    For best practices regarding training and finetuning, refer to the documentation in src/f5_tts/train.

    # Quick start finetuning via web interface
f5-tts_finetune-gradio

  10. How to contribute new F5-TTS models

    main

    If you have a pretrained or finetuned model, you can contribute it to the community by following these steps:

    1. Prepare your files: Ensure you have a model checkpoint (it is recommended to prune it to include only the ema_model_state_dict for easier inference) and the corresponding vocab file used for tokenization.
    2. Host the model: Upload your model files to a public Hugging Face model repository.
    3. Submit a PR: Create a pull request to add a new model card to src/f5_tts/infer/SHARED.md following the existing format.

  11. Run F5-TTS using Docker

    main

    You can use Docker to run F5-TTS without local dependency management. You can either build from the provided Dockerfile or pull the image from the GitHub Container Registry.

    To run the web interface (Gradio) via Docker, use the f5-tts_infer-gradio entrypoint.

    # Build from Dockerfile
docker build -t f5tts:v1 .

# Run from GitHub Container Registry (Web Interface)
docker container run --rm -it --gpus=all --mount 'type=volume,source=f5-tts,target=/root/.cache/huggingface/hub/' -p 7860:7860 ghcr.io/swivid/f5-tts:main f5-tts_infer-gradio --host 0.0.0.0

  12. Install evaluation dependencies

    main

    To install the necessary packages for running evaluations, use the [eval] extra with pip.

    Important: faster-whisper compatibility requirements: Depending on your CUDA and cuDNN versions, you must install a specific version of ctranslate2 to ensure compatibility:

    • CUDA 12 and cuDNN 9: pip install ctranslate2==4.5.0
    • CUDA 12 and cuDNN 8: pip install ctranslate2==4.4.0
    • CUDA 11 and cuDNN 8: pip install ctranslate2==3.24.0
    pip install -e .[eval]