TorToiSe TTS

Tortoise is a multi-speaker model that uses reference clips to guide speech generation. These clips determine the pitch, tone, speaking speed, and even speaking defects (like lisps or stutters) of the output.

Crucially, the reference clips also influence non-voice audio characteristics, such as volume, background noise, recording quality, and reverb. To achieve high-quality results, the quality of your reference clips is paramount.

Use Docker to set up an interactive environment. This requires an NVIDIA GPU and the NVIDIA Container Toolkit.

git clone https://github.com/neonbjb/tortoise-tts.git
cd tortoise-tts

docker build . -t tts

docker run --gpus all \
    -e TORTOISE_MODELS_DIR=/models \
    -v /mnt/user/data/tortoise_tts/models:/models \
    -v /mnt/user/data/tortoise_tts/results:/results \
    -v /mnt/user/data/.cache/huggingface:/root/.cache/huggingface \
    -v /root:/work \
    -it tts

On macOS 13+ with M1/M2 chips, you must install the nightly version of PyTorch. DeepSpeed is not supported on Apple Silicon (the --use_deepspeed flag is ignored). You may need to prepend PYTORCH_ENABLE_MPS_FALLBACK=1 to commands to handle operations not supported by MPS.

Using venv:

python3.10 -m venv .venv
source .venv/bin/activate
pip install numba inflect psutil
pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu
pip install transformers
git clone https://github.com/neonbjb/tortoise-tts.git
cd tortoise-tts
pip install .

For Windows users, using Conda is highly recommended to avoid dependency issues. Follow these steps in an Anaconda Prompt:

1. Create and activate a conda environment with required dependencies.
2. Install PyTorch with CUDA support.
3. Install transformers.
4. Clone the repository and install via setup.py.

Note: You may also need to install pysoundfile via conda install -c conda-forge pysoundfile on Windows.

conda create --name tortoise python=3.9 numba inflect
conda activate tortoise
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
conda install transformers=4.29.2
git clone https://github.com/neonbjb/tortoise-tts.git
cd tortoise-tts
python setup.py install

The quality of your reference clips profoundly impacts the output. Follow these guidelines to ensure high-quality speech generation:

What to avoid:

• Background noise: Avoid clips with background music, noise, or reverb (these were excluded from the training set).
• Speeches: Avoid formal speeches, as amplification systems often cause distortion.
• Phone calls: Avoid audio from phone conversations.
• Disfluencies: Avoid clips with excessive stuttering, stammering, or filler words like "uh" or "like".

What to look for:

• Desired Style: Choose clips that match the intended output style (e.g., if you want an audiobook style, use clips of the speaker reading a book).
• Text Diversity: While the specific text spoken in the clips doesn't matter, using clips with diverse text tends to perform better.

The repository includes several pre-packaged voices.

• High Quality: Use voices prepended with train_. These were part of the original training set and offer the best performance.
• Zero-shot Mimicking: Use the other provided voices to test Tortoise's ability to mimic voices it hasn't been specifically trained on.

You can install the stable version of TorToiSe directly from PyPI using pip.

pip install tortoise-tts

To add a custom voice, follow these steps:

1. Gather audio: Collect audio clips of the target speaker (e.g., from YouTube, audiobooks, or podcasts).
2. Segment clips: Cut the audio into segments of approximately 10 seconds each. You need at least 3 clips (5 is recommended for better results).
3. Format clips: Save clips as WAV files using floating point format and a 22,050 sample rate.
4. Organize files: Create a new subdirectory inside the voices/ directory.
5. Store clips: Place your formatted WAV clips into that new subdirectory.
6. Execute: Run Tortoise utilities using the --voice flag pointing to your subdirectory name.

Example command structure:

tortoise_utility_command --voice=my_custom_voice

tortoise_utility_command --voice=<your_subdirectory_name>

To install the most recent development version directly from the GitHub repository, use the following command:

pip install git+https://github.com/neonbjb/tortoise-tts

You can influence the tonality of the generated speech by including emotional cues in the prompt. Tortoise includes an automated redaction system that identifies text surrounded by brackets and removes it from the spoken output while using it to set the tone.

For example, the prompt [I am really sad,] Please feed me. will result in the model speaking only "Please feed me" but with a sad tonality.

To install Tortoise-TTS, first ensure you have the requirements installed. It is strongly recommended to use a GPU runtime, as inference on a CPU is extremely slow. You can install the dependencies via pip using the provided requirements.txt file.

# Follow instructions in README.md for Local Installation
!pip3 install -r requirements.txt
# or
# !python3 setup.py install