repository·dev·Indexed 25 days ago
https://github.com/idiap/coqui-ai-ttsA high-performance deep learning library for advanced Text-to-Speech (TTS) generation and voice conversion, supporting over 1100 languages. It provides tools for training, fine-tuning, and dataset curation, featuring spectrogram models (Tacotron, Glow-TTS), end-to-end models (XTTS, VITS, Bark), and various vocoders. The library offers both a Python API and a Command Line Interface for speech synthesis and voice cloning.
XTTS is a multi-lingual Text-to-Speech model capable of voice cloning using a short (approx. 3-second) audio clip. It supports cross-language voice cloning and multi-lingual speech generation.
Key Features:
Coqui TTS is a library for advanced Text-to-Speech generation, supporting over 1100 languages. It provides tools for training, fine-tuning, and dataset curation.
Core Capabilities:
dataset_analysis/ for curation.The Coqui TTS repository is organized into core model implementations, recipes, and utility directories.
TTS/)TTS/api.py: The primary Python API for interacting with the library.TTS/bin/: Contains the executable scripts and Command Line Interface (CLI) tools.TTS/tts/: Contains text-to-speech model definitions, including configs/ for model configurations, layers/ for layer definitions, and models/ for the model architectures themselves.TTS/vc/: Contains voice conversion models.TTS/vocoder/: Contains vocoder models.TTS/encoder/: Contains speaker encoder models..models.json: A list of available pretrained models.notebooks/: Jupyter Notebooks designed for model evaluation, parameter selection, and data analysis.recipes/: Contains training recipes for various models.pyproject.toml: Defines project metadata, dependencies, and configuration.Coqui TTS offers three primary ways to perform speech synthesis (inference):
tts command in your terminal for quick synthesis tasks.server.md for details).The project provides experimental implementations of several vocoder models that can be combined with other TTS models. Currently supported models include:
MelganMultiBand-MelganParallelWaveGANGAN-TTS (Discriminator Only)The Model API provides a standardized set of base classes that ensure your custom models are compatible with the following core components of the Coqui TTS ecosystem:
Trainer: For training and fine-tuning models.TTS.utils.synthesizer.Synthesizer: For high-level synthesis tasks.Overflow TTS is a neural transducer model for text-to-speech that combines Neural Hidden Markov Models (HMMs) with normalizing flows. This approach aims to provide a fully probabilistic model of durations and acoustics that can be trained using exact maximum likelihood.
Key advantages include:
The TTS.tts.utils.speakers.SpeakerManager is a utility class used to organize speaker-related data and information for TTS models. It is primarily used in multi-speaker models to manage speaker identities.
Key Behaviors:
SpeakerManager is initialized automatically when either use_speaker_embedding or use_d_vector_file is set in your model configuration.speaker_name field within your dataset.To use multi-speaker capabilities effectively, ensure your dataset is formatted with a speaker_name field for every entry.
For training vocoders, the project provides specialized dataset classes depending on the architecture being used:
GANDataset (from TTS.vocoder.datasets.gan_dataset): For training Generative Adversarial Network-based vocoders.WaveGradDataset (from TTS.vocoder.datasets.wavegrad_dataset): For training WaveGrad models.WaveRNNDataset (from TTS.vocoder.datasets.wavernn_dataset): For training WaveRNN models.Coqui uses the coqpit-config package (via Coqpit) for configuration management. It leverages Python dataclasses to provide static type checking and serialization.
Key features include:
MISSING as a default value to ensure an error is raised if the field is not explicitly provided.dict, List[List], and List[Union[...]] using field(default_factory=...).check_values method within your configuration class to enforce constraints (e.g., min/max values) using check_argument.from dataclasses import asdict, dataclass, field
from typing import List, Union
from coqpit.coqpit import MISSING, Coqpit, check_argument
@dataclass
class SimpleConfig(Coqpit):
val_a: int = 10
val_b: int = None
val_d: float = 10.21
val_c: str = "Coqpit is great!"
vol_e: bool = True
# mandatory field
val_k: int = MISSING
# optional field
val_dict: dict = field(default_factory=lambda: {"val_aa": 10, "val_ss": "This is in a dict."})
# list of list
val_listoflist: List[List] = field(default_factory=lambda: [[1, 2], [3, 4]])
val_listofunion: List[List[Union[str, int, bool]]] = field(
default_factory=lambda: [[1, 3], [1, "Hi!"], [True, False]]
)
def check_values(self):
"""Check config fields"""
c = asdict(self)
check_argument("val_a", c, restricted=True, min_val=10, max_val=2056)
check_argument("val_b", c, restricted=True, min_val=128, max_val=4058, allow_none=True)
check_argument("val_c", c, restricted=True)To add voice cloning capabilities to a custom model, the model class must inherit from TTS.utils.voices.CloningMixin in addition to its base TTS or VC class.
Developers must implement a model-specific _clone_voice() method that returns speaker embeddings and model-specific metadata. The CloningMixin then handles the automatic caching of these voices. In the model's synthesize() method, use CloningMixin.clone_voice to access the voice data.