DOCSEARCHbyAlgolia
Submit Docs

Orpheus TTS Documentation

repository·main·Indexed 27 days ago

https://github.com/canopyai/orpheus-tts

An open-source text-to-speech system built on a Llama-3b backbone featuring human-like speech, zero-shot voice cloning, and guided emotion/intonation control. The system supports streaming inference via the orpheus-speech package and a CPU-based backend called orpheus-cpp. It includes tools for pretraining and finetuning using Transformers and accelerate, as well as deployment options for Baseten with fp8 and fp16 precision.

Tokens
3K
Snippets
10
Records
17
Agent score
92%

What's inside Orpheus TTS

  1. Install orpheus-cpp for CPU-based streaming inference

    main

    To perform streaming audio inference without a GPU, use the orpheus-cpp backend. This requires installing orpheus-cpp and a CPU-optimized version of llama-cpp-python depending on your operating system.

    Installation Steps

    1. Install orpheus-cpp

      pip install orpheus-cpp

    2. Install llama-cpp-python

      • Linux/Windows (CPU):
        pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
      • MacOS (Apple Silicon/Metal):
        pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal
    pip install orpheus-cpp
# For Linux/Windows
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
# For MacOS Apple Silicon
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal

  3. Deploy Orpheus TTS on Baseten

    main

    You can deploy Orpheus TTS to production using Baseten.

  5. Optimize text-to-speech batch ratios during pretraining

    main

    The model's ability to speak naturally and empathetically is improved by maintaining semantic understanding through text training. You can adjust the ratio of text batches to speech batches to control the model's capabilities:

    • End-to-end speech model (retaining text ability): Start with a text batch to speech batch ratio of 2:1, then gradually decrease to 1:1 during training.
    • TTS-only model: Start with a 1:1 ratio and gradually decrease to 0:1 during training.

  6. Finetune Orpheus models

    main

    Finetuning Orpheus is analogous to tuning an LLM using Transformers. High-quality results are typically seen after ~50 examples, but 300 examples per speaker is recommended for best results.

    Workflow:

    1. Prepare a Hugging Face dataset in the required format.
    2. Use the provided preprocessing notebook to prepare the data.
    3. Modify finetune/config.yaml with your dataset and training properties.
    4. Run the training script using accelerate.
    pip install transformers datasets wandb trl flash_attn torch
huggingface-cli login <enter your HF token>
wandb login <wandb token>
accelerate launch train.py
    pip install transformers datasets wandb trl flash_attn torch
huggingface-cli login <enter your HF token>
wandb login <wandb token>
accelerate launch train.py

  7. Install Orpheus TTS

    main

    To use Orpheus TTS for streaming inference, clone the repository and install the orpheus-speech package via pip. Note that orpheus-speech uses vllm under the hood. If you encounter issues with vllm (specifically related to a version released around March 18th), you may need to revert to vllm==0.7.3 after installation.

    git clone https://github.com/canopyai/Orpheus-TTS.git
cd Orpheus-TTS && pip install orpheus-speech
# If bugs occur, run:
pip install vllm==0.7.3
    git clone https://github.com/canopyai/Orpheus-TTS.git
cd Orpheus-TTS && pip install orpheus-speech

  8. Install dependencies for pretraining

    main

    To set up the environment for pretraining, install the required packages using pip. Note that you may need to experiment with different versions of flash_attn to ensure compatibility with your specific torch, cuda, and python versions.

    pip install transformers trl wandb flash_attn datasets torch

  10. Resolve KV Cache or max_model_len errors

    main

    If you encounter a KV cache error or find that the max_model_len property does not exist after installing via PyPI, you should use the local package from the cloned repository instead. This ensures you are using the most recent fixes.

    import sys
sys.path.insert(0, 'orpheus_tts_pypi')
from orpheus_tts import OrpheusModel
    import sys
sys.path.insert(0, 'orpheus_tts_pypi')
from orpheus_tts import OrpheusModel

  11. Perform Streaming Inference with OrpheusModel

    main

    Use the OrpheusModel class to generate speech. The generate_speech method returns a generator of audio chunks, allowing for real-time streaming output. You can specify the model_name (e.g., canopylabs/orpheus-tts-0.1-finetune-prod), the prompt text, and the target voice (e.g., tara).

    To save the streamed output to a WAV file, use the wave module to write the incoming audio chunks.

    from orpheus_tts import OrpheusModel
import wave
import time

model = OrpheusModel(model_name ="canopylabs/orpheus-tts-0.1-finetune-prod", max_model_len=2048)
prompt = '''Man, the way social media has, um, completely changed how we interact is just wild, right? Like, we're all connected 24/7 but somehow people feel more alone than ever. And don't even get me started on how it's messing with kids' self-esteem and mental health and whatnot.'''

start_time = time.monotonic()
syn_tokens = model.generate_speech(
   prompt=prompt,
   voice="tara",
   )

with wave.open("output.wav", "wb") as wf:
   wf.setnchannels(1)
   wf.setsampwidth(2)
   wf.setframerate(24000)

   total_frames = 0
   chunk_counter = 0
   for audio_chunk in syn_tokens: # output streaming
      chunk_counter += 1
      frame_count = len(audio_chunk) // (wf.getsampwidth() * wf.getnchannels())
      total_frames += frame_count
      wf.writeframes(audio_chunk)
   duration = total_frames / wf.getframerate()

   end_time = time.monotonic()
   print(f"It took {end_time - start_time} seconds to generate {duration:.2f} seconds of audio")

  12. Perform streaming TTS inference with OrpheusCpp

    main

    Use the OrpheusCpp class to stream text-to-speech audio synchronously. The stream_tts_sync method yields tuples containing the sample rate (sr) and the audio chunk. You can pass configuration via the options dictionary, such as specifying a voice_id.

    from scipy.io.wavfile import write
from orpheus_cpp import OrpheusCpp
import numpy as np

# Initialize with desired language
orpheus = OrpheusCpp(verbose=False, lang="en")

text = "I really hope the project deadline doesn't get moved up again."
buffer = []

# Stream chunks synchronously
for i, (sr, chunk) in enumerate(orpheus.stream_tts_sync(text, options={"voice_id": "tara"})):
    buffer.append(chunk)
    print(f"Generated chunk {i}")

# Concatenate and save to file
full_audio = np.concatenate(buffer, axis=1)
write("output.wav", 24_000, full_audio)