OuteTTS Documentation

repository·main·Indexed 23 days ago

OuteTTS is an experimental text-to-speech system (version 0.0.1) that uses a pure language modeling approach to generate speech. It supports multiple backends including llama.cpp, Hugging Face Transformers, EXL2, and VLLM. The library allows for high-quality speech generation using speaker profiles, custom audio-based speaker creation, and various quantization and hardware acceleration options for CPU, NVIDIA GPUs, AMD GPUs, Vulkan, and Apple Silicon.

Tokens: 4.3K
Snippets: 10
Records: 20
Agent score: 80%

What's inside OuteTTS

Prepare SFT data for Speaker Completion
main
Speaker Completion data allows the model to adapt to a specific speaker's style by providing a portion of the speaker's audio as context.
Implementation Details:
- The input should include a portion of the speaker's audio (typically 5-10 seconds).
- It is recommended to use varying segment lengths rather than a fixed length to improve robustness.
Input Format: Includes the text prompt followed by a segment of the speaker's audio tokens.
```
<|im_start|>
<|text_start|>text<|period|><|text_end|>
<|audio_start|>
[Partial Audio Tokens]
```
Target Format: The remaining audio tokens required to complete the sequence.
```
[Remaining Audio Tokens]
<|audio_end|>
<|im_end|>
```
```
<|im_start|>
<|text_start|>this<|space|>is<|space|>a<|space|>test<|period|><|text_end|>
<|audio_start|>
this<|t_0.15|><|27|><|1789|><|379|><|1236|><|1465|><|1326|><|1584|><|889|><|183|><|1283|><|794|><|space|>
is<|t_0.09|><|1281|><|903|><|1521|><|319|><|230|><|1533|><|906|><|space|>
```
https://github.com/edwko/outetts/blob/main/examples/training/OuteTTS-0.3/train.md
Prepare SFT data for Input Completion
main
Input Completion data is used to train the model to predict audio completions based on a given text input. The data follows a specific prompt structure using special tokens.
Input Format:
```
<|im_start|>
<|text_start|>text<|space|>here<|period|><|text_end|>
<|audio_start|>
```
Target Format: The target contains the text tokens followed by the corresponding audio tokens (e.g., <|t_0.15|><|27|>...) and ends with the audio end tokens.
```
text<|t_0.15|><|27|>...<|space|>
<|audio_end|>
<|im_end|>
```
```
<|im_start|>
<|text_start|>this<|space|>is<|space|>a<|space|>test<|period|><|text_end|>
<|audio_start|>
```
https://github.com/edwko/outetts/blob/main/examples/training/OuteTTS-0.3/train.md
Install OuteTTS via Pip
main
OuteTTS installs llama.cpp Python bindings by default. You must choose an installation command based on your hardware to ensure the correct backend is compiled.
CPU (Transformers + llama.cpp CPU)
```
pip install outetts --upgrade
```
NVIDIA GPUs (Transformers + llama.cpp CUDA)
```
CMAKE_ARGS="-DGGML_CUDA=on" pip install outetts --upgrade
```
AMD GPUs (Transformers + llama.cpp ROCm/HIP)
```
CMAKE_ARGS="-DGGML_HIPBLAS=on" pip install outetts --upgrade
```
Vulkan (Cross-platform GPU)
```
CMAKE_ARGS="-DGGML_VULKAN=on" pip install outetts --upgrade
```
Apple Silicon/Mac (Transformers + llama.cpp Metal)
```
CMAKE_ARGS="-DGGML_METAL=on" pip install outetts --upgrade
```
https://github.com/edwko/outetts/blob/main/README.md

Basic Usage of OuteTTS

main

To use OuteTTS, initialize an outetts.Interface with a ModelConfig, load a speaker profile, and then call interface.generate with a GenerationConfig.

Note: Currently, only one default English voice is available for testing. You can create custom speaker profiles from audio files using interface.create_speaker("path/to/audio.wav").

import outetts

# Initialize the interface
interface = outetts.Interface(
    config=outetts.ModelConfig.auto_config(
        model=outetts.Models.VERSION_1_0_SIZE_1B,
        # For llama.cpp backend
        backend=outetts.Backend.LLAMACPP,
        quantization=outetts.LlamaCppQuantization.FP16
        # For transformers backend
        # backend=outetts.Backend.HF,
    )
)

# Load the default speaker profile
speaker = interface.load_default_speaker("EN-FEMALE-1-NEUTRAL")

# Generate speech
output = interface.generate(
    config=outetts.GenerationConfig(
        text="Hello, how are you doing?",
        speaker=speaker,
    )
)

# Save to file
output.save("output.wav")

Create a dataset for OuteTTS v0.3 fine-tuning
main
To fine-tune OuteTTS v0.3, you must first convert your raw data into a specific Parquet format. Use the processing script located at examples/training/OuteTTS-0.3/data_creation_example.py to automate this.
Requirements for raw data:
- Format: Parquet files.
- Required fields:
  transcript: A string containing the text.
  audio: Audio bytes stored in the bytes format.
The script will output Parquet files containing the processed data with full prompts.
https://github.com/edwko/outetts/blob/main/examples/training/OuteTTS-0.3/train.md
Best Practices for Audio Generation
main
To achieve high-quality results with OuteTTS, follow these guidelines:
- Optimal Audio Length: Aim for approximately 42 seconds (about 8,192 tokens) per generation. For best results, stay under 7,000 tokens.
- Multilingual Use: Create speaker profiles in the target language to ensure correct tone, accent, and linguistic features. Using a speaker from one language (e.g., Japanese) for another (e.g., English) may result in the model retaining the original accent.
- Speaker Reference Quality: Ensure the reference audio is free from clipping, excessive loudness, or unusual vocal features, as the DAC reconstruction is lossy.
https://github.com/edwko/outetts/blob/main/README.md
Manage Speaker Profiles
main
OuteTTS allows you to use default speakers or create custom ones from audio files.
Using Default Speakers
Use interface.load_default_speaker(name) to load a built-in profile. You can see all available speakers by calling interface.print_default_speakers().
Creating Custom Speakers
To create a profile from an audio file, use interface.create_speaker(audio_path). This uses Whisper for transcription and an audio codec to generate the profile.
Best Practice: Save your custom speaker to a JSON file using interface.save_speaker(speaker, "path.json") so you can reuse it without re-processing the audio.
Loading Custom Speakers
Load a previously saved profile using interface.load_speaker("path.json").
Advanced Speaker Creation
For older models, you can provide a transcript manually:
```
speaker = interface.create_speaker(
    audio_path="path/to/audio.wav",
    transcript="What is being said",
    whisper_model="turbo",
    whisper_device="cuda"
)
```
```
# Create and save a custom speaker
speaker = interface.create_speaker("path/to/audio.wav")
interface.save_speaker(speaker, "my_speaker.json")

# Load it later
speaker = interface.load_speaker("my_speaker.json")
```
https://github.com/edwko/outetts/blob/main/docs/interface_usage.md
Train the OuteTTS v0.3 model
main
OuteTTS is built on a language model architecture. Because of this, you can use any standard language model training framework to perform Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO).
Supported Frameworks:
https://github.com/edwko/outetts/blob/main/examples/training/OuteTTS-0.3/train.md
Initialize the OuteTTS Interface
main
To use OuteTTS, you must first initialize the outetts.Interface with a ModelConfig. You can use the recommended auto_config method to quickly set up a model and backend, or provide a manual ModelConfig for fine-grained control over paths, devices, and backend-specific settings.
Recommended Setup (auto_config)
Use outetts.ModelConfig.auto_config() to specify the model version and backend (e.g., LLAMACPP or HF).
Manual Setup
For advanced users, outetts.ModelConfig allows specifying model_path, tokenizer_path, device, dtype, and additional_model_config for passing parameters like attn_implementation directly to the backend.
```
import outetts

# Recommended initialization
interface = outetts.Interface(
    config=outetts.ModelConfig.auto_config(
        model=outetts.Models.VERSION_1_0_SIZE_1B,
        backend=outetts.Backend.LLAMACPP,
        quantization=outetts.LlamaCppQuantization.FP16
    )
)
```
https://github.com/edwko/outetts/blob/main/docs/interface_usage.md

Perform Batch Speech Generation

main

For high-throughput tasks using backends like VLLM, EXL2ASYNC, or LLAMACPP_ASYNC_SERVER, use GenerationType.BATCH. This mode processes multiple text chunks in parallel.

When using these backends, BATCH is often automatically selected. You can tune performance using max_batch_size and dac_decoding_chunk within the GenerationConfig.

Note for LLAMACPP_ASYNC_SERVER: You must provide the server_host in the GenerationConfig (e.g., server_host="http://localhost:8000").

from outetts import Interface, ModelConfig, GenerationConfig, Backend, GenerationType

interface = Interface(
    ModelConfig(
        model_path="OuteAI/Llama-OuteTTS-1.0-0.6B-FP8",
        tokenizer_path="OuteAI/Llama-OuteTTS-1.0-0.6B",
        backend=Backend.VLLM
    )
)

speaker = interface.load_default_speaker("EN-FEMALE-1-NEUTRAL")

output = interface.generate(
    GenerationConfig(
        text="This text will be processed in batches.",
        speaker=speaker,
        generation_type=GenerationType.BATCH,
        max_batch_size=32,
        dac_decoding_chunk=2048
    )
)
output.save("output_batch.wav")

Use Hugging Face backend with Flash Attention

main

To use the Hugging Face (outetts.Backend.HF) backend with advanced optimizations like Flash Attention 2, initialize the outetts.Interface with a ModelConfig. You can pass hardware-specific optimizations via the additional_model_config dictionary. This is useful for high-performance inference on compatible CUDA devices.

import outetts
import torch

interface = outetts.Interface(
    config=outetts.ModelConfig(
        model_path="OuteAI/Llama-OuteTTS-1.0-1B",
        tokenizer_path="OuteAI/Llama-OuteTTS-1.0-1B",
        interface_version=outetts.InterfaceVersion.V3,
        backend=outetts.Backend.HF,
        additional_model_config={
            "attn_implementation": "flash_attention_2"  # Enable flash attention if compatible
        },
        device="cuda",
        dtype=torch.bfloat16
    )
)

Pass Custom Settings to Backends
main
OuteTTS provides two ways to pass backend-specific parameters that are not part of the standard API.
1. Model Initialization (additional_model_config)
Pass parameters to the model loading function via ModelConfig.additional_model_config. This is useful for settings like attn_implementation or device_map.
2. Generation (additional_gen_config)
Pass parameters to the model's generation function via GenerationConfig.additional_gen_config. This allows using backend-specific parameters like frequency_penalty or presence_penalty.
```
# Example: Custom generation parameters
output = interface.generate(
    config=outetts.GenerationConfig(
        text="Hello",
        speaker=speaker,
        additional_gen_config={
            "frequency_penalty": 1.0,
            "presence_penalty": 0.5,
        }
    )
)
```
https://github.com/edwko/outetts/blob/main/docs/interface_usage.md

OuteTTS Documentation

What's inside OuteTTS

Prepare SFT data for Speaker Completion

Prepare SFT data for Input Completion

Install OuteTTS via Pip

CPU (Transformers + llama.cpp CPU)

NVIDIA GPUs (Transformers + llama.cpp CUDA)

AMD GPUs (Transformers + llama.cpp ROCm/HIP)

Vulkan (Cross-platform GPU)

Apple Silicon/Mac (Transformers + llama.cpp Metal)

Basic Usage of OuteTTS

Create a dataset for OuteTTS v0.3 fine-tuning

Best Practices for Audio Generation

Manage Speaker Profiles

Using Default Speakers

Creating Custom Speakers

Loading Custom Speakers

Advanced Speaker Creation

Train the OuteTTS v0.3 model

Initialize the OuteTTS Interface

Recommended Setup (auto_config)

Manual Setup

Perform Batch Speech Generation

Use Hugging Face backend with Flash Attention

Pass Custom Settings to Backends

1. Model Initialization (`additional_model_config`)

2. Generation (`additional_gen_config`)

OuteTTS Documentation

What's inside OuteTTS

CPU (Transformers + llama.cpp CPU)

NVIDIA GPUs (Transformers + llama.cpp CUDA)

AMD GPUs (Transformers + llama.cpp ROCm/HIP)

Vulkan (Cross-platform GPU)

Apple Silicon/Mac (Transformers + llama.cpp Metal)

Using Default Speakers

Creating Custom Speakers

Loading Custom Speakers

Advanced Speaker Creation

Recommended Setup (auto_config)

Manual Setup

1. Model Initialization (additional_model_config)

2. Generation (additional_gen_config)

1. Model Initialization (`additional_model_config`)

2. Generation (`additional_gen_config`)