DOCSEARCHbyAlgolia
Submit Docs

Coqui TTS

repository·dev·Indexed 13 days ago

https://github.com/coqui-ai/tts

A comprehensive library for advanced Text-to-Speech generation supporting over 1100 languages. It provides tools for training, fine-tuning, and dataset curation, featuring model implementations such as XTTS, VITS, YourTTS, Tortoise, and Bark, as well as vocoders like HiFiGAN and MelGAN.

Tokens
47.4K
Snippets
150
Records
202
Agent score
98%

What's inside Coqui TTS

  1. Overview of 🐸TTS Features and Model Types

    dev

    🐸TTS is a library for advanced Text-to-Speech generation, supporting over 1100 languages. It provides tools for training, fine-tuning, and dataset curation.

    Core Capabilities

    • High-performance Deep Learning models: Includes Text2Spec models, Speaker Encoders, and Vocoders.
    • Training Utilities: Fast model training with detailed logs via terminal and Tensorboard.
    • Multi-speaker Support: Capability to handle multiple voices.
    • Trainer API: An efficient, flexible, and lightweight API for model training.
    • Dataset Tools: Utilities under dataset_analysis for curating Text2Speech datasets.

    Model Implementations

    • Spectrogram Models: Tacotron, Tacotron2, Glow-TTS, FastSpeech, etc.
    • End-to-End Models: XTTS, VITS, YourTTS, Tortoise, and Bark.
    • Vocoders: MelGAN, HiFiGAN, WaveGrad, etc.
    • Voice Conversion: FreeVC.

  2. Overview of XTTS-v2

    dev

    XTTS-v2 is a high-performance Text-to-Speech model designed for voice cloning and multi-lingual speech generation. It can clone a voice using as little as a 3-second audio clip and supports cross-language voice cloning.

    Key Features:

    • Voice cloning (single or multiple audio references).
    • Cross-language voice cloning.
    • Multi-lingual support (16 languages).
    • 24khz sampling rate.
    • Streaming inference with < 200ms latency.
    • Fine-tuning support.

    Supported Languages: English (en), Spanish (es), French (fr), German (de), Italian (it), Portuguese (pt), Polish (pl), Turkish (tr), Russian (ru), Dutch (nl), Czech (cs), Arabic (ar), Chinese (zh-cn), Japanese (ja), Hungarian (hu), and Korean (ko).

  3. Available Mozilla TTS Vocoder implementations

    dev

    The project provides experimental implementations of several vocoder models that can be combined with other TTS models. Currently supported models include:

    • Melgan
    • MultiBand-Melgan
    • ParallelWaveGAN
    • GAN-TTS (Discriminator Only)

  4. Use Tortoise for expressive TTS and voice cloning

    dev

    Tortoise is a highly expressive TTS system capable of impressive voice cloning. It uses a GPT-like autoregressive acoustic model to convert text to acoustic tokens, a diffusion model to convert tokens to mel-spectrograms, and a Univnet vocoder for final audio.

    Note: Tortoise is significantly slower than parallel models like VITS due to its autoregressive and diffusion-based architecture.

  5. What is the VITS model architecture?

    dev

    VITS (Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech) is an End-to-End TTS model that combines an encoder and a vocoder into a single architecture. It utilizes SOTA deep learning techniques including GANs, VAE, and Normalizing Flows.

    Key characteristics:

    • Architecture: A combination of a GlowTTS encoder and a HiFiGAN vocoder.
    • Alignment: It does not require external alignment annotations; it learns text-to-audio alignment using Monotonic Alignment Search (MAS).
    • Performance: It is a feed-forward model capable of a ~67.12 real-time factor on a GPU.
    • YourTTS: A specific implementation of VITS that serves as a multi-speaker and multi-lingual backbone. It supports voice conversion, zero-shot speaker adaptation, and can learn new languages or voices from approximately 1 minute of audio, making it suitable for low-resource language training.

  6. What is Overflow TTS

    dev

    Overflow TTS is a neural HMM (Hidden Markov Model) based transducer for sequence-to-sequence text-to-speech modeling. It combines classic statistical speech synthesis with modern neural TTS by using normalizing flows to describe non-Gaussian speech acoustics.

    Key characteristics include:

    • Robustness: Less prone to 'gibberish' output caused by neural attention failures.
    • Efficiency: Requires less data and fewer training updates compared to some modern neural models.
    • Probabilistic Modeling: Provides a fully probabilistic model of durations and acoustics, allowing for training via exact maximum likelihood.
    • Prosody: Integrates autoregression to better model long-range dependencies like utterance-level prosody.

  7. Use Vocoder Datasets for GAN, WaveGrad, or WaveRNN training

    dev

    For training vocoders (models that convert spectrograms to audio), use the specialized dataset classes depending on the architecture being implemented:

    • GANDataset: For Generative Adversarial Network-based vocoders.
    • WaveGradDataset: For WaveGrad-based models.
    • WaveRNNDataset: For WaveRNN-based models.
    from TTS.vocoder.datasets.gan_dataset import GANDataset
from TTS.vocoder.datasets.wavegrad_dataset import WaveGradDataset
from TTS.vocoder.datasets.wavernn_dataset import WaveRNNDataset

  8. Understand the TTS configuration hierarchy

    dev

    In TTS, every model requires a configuration class that exposes all values necessary for its lifetime, including architecture, hyperparameters, training, and inference settings.

    While it is recommended to use a single unified configuration class for ease of bookkeeping and reproducibility, the project follows a hierarchical structure where a top-level ModelConfig() aggregates several specialized configuration objects:

    • ModelArgs(): Model class arguments.
    • BaseDatasetConfig(): Specific to TTS models.
    • BaseXModelConfig(): Generic fields shared by tts and vocoder models.
      • BaseTrainingConfig(): Trainer-specific fields.
      • BaseAudioConfig(): Audio processing fields.

    Pre-defined model configurations are located in TTS/<model_class>/configs/.

    ModelConfig()
     |
     | -> ...         # model specific configurations
     | -> ModelArgs()           # model class arguments
     | -> BaseDatasetConfig()    # only for tts models
     | -> BaseXModelConfig()    # Generic fields for `tts` and `vocoder` models.
                |
                | -> BaseTrainingConfig()   # trainer fields
                | -> BaseAudioConfig()      # audio processing fields

  9. How multi-speaker training works

    dev

    Multi-speaker training follows the same workflow as single-speaker training but requires specific configuration changes:

    1. Configuration: Define parameters to determine if the model uses a speaker-embedding layer or pre-computed d-vectors.
    2. SpeakerManager: You must initiate a SpeakerManager instance and pass it to the model.
    3. D-vectors: If using d-vectors, they must be pre-computed using the SpeakerEncoder before training.

    Example recipes for multi-speaker datasets (like VCTK) can be found in TTS/recipes/vctk/.

  10. Understand the Forward TTS model architecture

    dev

    The ForwardTTS model is a general feed-forward Text-to-Speech implementation. Its architecture is flexible and can be configured by specifying different encoder and decoder networks.

    Key characteristics include:

    • Training Modes: It can be trained using pre-computed durations (e.g., from a pre-trained Tacotron model) or via an alignment network that learns text-to-audio alignment directly from input data.
    • Available Architectures:
      • FastSpeech: Uses Feed Forward Transformer (FFT) modules for both encoder and decoder.
      • FastPitch: Based on the FastSpeech architecture but conditioned on fundamental frequency (f0) contours to enable more expressive speech.
      • SpeedySpeech: Replaces Transformers with Residual Convolution layers to create a more compute-friendly model.
      • FastSpeech2 (Planned): An extension of FastPitch that incorporates spectral energy values.

  11. Configure ForwardTTS via architecture-specific configs

    dev

    To use specific feed-forward architectures, you must use their corresponding configuration classes. These classes define the hyperparameters and structural settings for the model:

    • FastPitchConfig: For configuring FastPitch models (FFT-based with f0 conditioning).
    • SpeedySpeechConfig: For configuring SpeedySpeech models (Residual Convolution-based).
    • FastSpeechConfig: For configuring standard FastSpeech models (FFT-based).

    These configurations are used to instantiate the ForwardTTS model with the desired architectural behavior.

  12. Use the Speaker Encoder for voice and speaker embedding

    dev

    The Speaker Encoder is an implementation of the d-vector approach (based on arXiv:1710.10467). It can be used to generate embeddings for both multi-speaker and single-speaker TTS datasets. These embeddings (d-vectors) can be used for voice cloning or speaker identification tasks.

    To use the encoder, you can either train it on your own dataset or download a pretrained model from the Released Models page.