DOCSEARCHbyAlgolia
Submit Docs

GPT-SoVITS: Few-Shot Voice Conversion and Text-to-Speech

repository·main·Indexed 13 days ago

https://github.com/rvc-boss/gpt-sovits

A powerful few-shot voice conversion and text-to-speech (TTS) system capable of creating realistic voice clones with as little as 5 seconds of audio. It supports English, Chinese, Japanese, Korean, and Cantonese, and includes an integrated WebUI for zero-shot and few-shot TTS, voice/accompaniment separation, and automatic speech recognition (ASR) using Fun-ASR-Nano, SenseVoice, and Faster Whisper.

Tokens
14.2K
Snippets
47
Records
95
Agent score
95%

What's inside GPT-SoVITS

  1. Overview of GPT-SoVITS Features

    main

    GPT-SoVITS is a powerful few-shot voice conversion and speech synthesis WebUI. Key capabilities include:

    • Zero-shot TTS: Instant text-to-speech conversion using only a 5-second audio sample.
    • Few-shot TTS: Fine-tune models with as little as 1 minute of training data to improve similarity and realism.
    • Cross-lingual Support: Inference in languages different from the training set, including English, Japanese, Korean, Cantonese, and Chinese.
    • Integrated WebUI Tools: Includes voice/accompaniment separation, automatic training set segmentation, multi-language Automatic Speech Recognition (ASR) using Fun-ASR-Nano, SenseVoice, and FunASR, and text annotation tools to assist in dataset creation.

  2. Overview of F5-TTS Backbone Architectures

    main

    The f5_tts module provides three distinct backbone architectures for text-to-speech modeling, each with different structural characteristics for handling text embeddings, noise, and conditioning:

    1. unett.py (Flat UNet Transformer): Follows the structure of E2-TTS and Voicebox papers but utilizes rotary positional embeddings. It supports absolute positional embeddings and ConvNextV2 blocks for embedded text prior to concatenation.
    2. dit.py (AdaLN-Zero DiT): A Diffusion Transformer using AdaLN-Zero. It conditions on embedded timesteps. The input is a concatenation of noised_input, masked_cond, and embedded_text, followed by a linear projection. It supports absolute positional embeddings, ConvNextV2 blocks for text, and optional long skip connections from the first to the last layer.
    3. mmdit.py (SD3 Structure): Based on the Stable Diffusion 3 architecture. It uses timestep conditioning with a dual-stream approach:
      • Left stream: Processes text embeddings with absolute positional embeddings.
      • Right stream: Processes concatenated masked_cond and noised_input using the same convolutional positional embeddings as the unett architecture.

  3. Use GPT-SoVITS V2Pro Series Models

    main
    The V2Pro series models were introduced in June 2025. Key features include support for torchscript export for V4 models and optimized precision auto-detection. Note that earlier versions of parallel inference had bugs with V2Pro support, so ensure you are using the latest release to avoid issues with model recognition or numerical overflows during ge extraction.

  4. Supported Model Series: V4 and V2Pro

    main

    GPT-SoVITS supports multiple model series for inference and training:

    • V4 Model: Introduced in April 2025. Supports parallel inference and has improved asynchronous model loading logic.
    • V2Pro Series: Introduced in June 2025. Includes V2Pro and V2ProPlus models. These models support parallel inference and can be exported with TorchScript.

    When using TTS.py, ensure you are using a version that recognizes these model names, as older versions may fail to identify v2Pro or v2ProPlus.

  5. Use GPT-SoVITS V4 Models

    main
    GPT-SoVITS V4 models were introduced in April 2025. These models support parallel inference and have been optimized for performance. Users should ensure their environment is updated to support V4 features, including improved model version parameter passing and compatibility with updated librosa and Numpy/Numba versions.

  6. Enable single precision for specific GPUs or CPU inference

    main
    For GPUs that do not support half precision (such as the NVIDIA 16 series) or when performing inference on a CPU, the system may automatically force single precision to prevent errors. If you encounter issues with half precision training on older hardware, ensure single precision is used.

  8. Inference on macOS and non-CUDA systems

    main

    GPT-SoVITS includes optimizations for non-GPU environments:

    • macOS: Supports CPU training and uses CPU instead of MPS for inference to improve performance.
    • Non-CUDA systems: Faster Whisper ASR will automatically switch to CPU inference if no CUDA device is detected.

  9. Format the TTS annotation .list file

    main

    For training, you must provide a .list annotation file where each line follows a specific pipe-delimited format. This file maps audio files to their metadata.

    Format: vocal_path|speaker_name|language|text

    Supported Language Keys:

    • zh: Chinese
    • ja: Japanese
    • en: English
    • ko: Korean
    • yue: Cantonese

    Example: D:\GPT-SoVITS\xxx/xxx.wav|xxx|en|I like playing Genshin.

    D:\GPT-SoVITS\xxx/xxx.wav|xxx|en|I like playing Genshin.

  12. Optimize BigVGAN inference speed

    main

    When instantiating the BigVGAN model for inference, you can significantly increase speed by enabling the CUDA kernel. This is particularly effective on NVIDIA GPUs (e.g., A100, RTX 3080).

    Set use_cuda_kernel=True during model instantiation to achieve much higher real-time factors compared to standard PyTorch implementations.