Chatterbox Streaming

To perform GRPO fine-tuning:

1. Place raw .wav files of the target speaker in a folder named audio_data.
2. Run grpo.py.
3. Configure training parameters (batch size, epochs, learning rate) at the top of grpo.py.

Requirements:

• CUDA GPU with at least 12GB VRAM.
• Training progress is monitored via grpo_training_metrics.png.

To install the package in a new Python 3.10 virtual environment, use the following commands:

python3.10 -m venv .venv
source .venv/bin/activate
pip install chatterbox-streaming

Use exaggeration and cfg_weight to control the style of the generated speech:

• General Use: Use default settings (exaggeration=0.5, cfg_weight=0.5).
• Fast Speakers: If the reference speaker is fast, lower cfg_weight to approximately 0.3 to improve pacing.
• Dramatic/Expressive Speech: Increase exaggeration to 0.7 or higher. To prevent the increased speed caused by high exaggeration, lower cfg_weight (e.g., ~0.3) to maintain slower, more deliberate pacing.

To perform LoRA fine-tuning:

1. Place raw .wav files of the target speaker in a folder named audio_data.
2. Run lora.py.
3. Configure training parameters (batch size, epochs, learning rate) at the top of lora.py.

Requirements:

• CUDA GPU with at least 18GB VRAM (depending on dataset/params).
• Training progress is monitored via training_metrics.png.
• To use a checkpoint, run loadandmergecheckpoint.py (ensure R and Alpha values match training).

The internal punc_norm function cleans up text (especially text coming from LLMs) before tokenization to ensure compatibility with the training data. It performs the following:

1. Capitalization: Ensures the first letter is uppercase.
2. Whitespace: Collapses multiple spaces into a single space.
3. Punctuation Replacement: Converts uncommon or LLM-style punctuation to standard forms:
   • ... or … $\rightarrow$ ,
   • : $\rightarrow$ ,
   • - $\rightarrow$ ,
   • ; $\rightarrow$ ,
   • “/” $\rightarrow$ "
   • ‘/’ $\rightarrow$ '
4. Sentence Completion: If the text does not end with a sentence ender (., !, ?, -, ,), a full stop (.) is appended.

For low-latency applications, use model.generate_stream(text, ...) which yields audio_chunk and metrics objects. This allows you to play audio chunks immediately as they are generated. You can combine all chunks into a single tensor using torch.cat(audio_chunks, dim=-1) for final saving.

import torchaudio as ta
import torch
from chatterbox.tts import ChatterboxTTS

model = ChatterboxTTS.from_pretrained(device="cuda")
text = "Welcome to the world of streaming text-to-speech!"

audio_chunks = []
for audio_chunk, metrics in model.generate_stream(text):
    audio_chunks.append(audio_chunk)
    # Play audio_chunk immediately here for real-time playback
    print(f"Generated chunk {metrics.chunk_count}, RTF: {metrics.rtf:.3f}")

final_audio = torch.cat(audio_chunks, dim=-1)
ta.save("streaming_output.wav", final_audio, model.sr)

Use ChatterboxTTS.from_pretrained(device="cuda") to load the model and .generate(text) to produce audio. You can also perform voice cloning by providing an audio_prompt_path to a reference .wav file.

import torchaudio as ta
from chatterbox.tts import ChatterboxTTS

model = ChatterboxTTS.from_pretrained(device="cuda")
text = "Ezreal and Jinx teamed up with Ahri, Yasuo, and Teemo to take down the enemy's Nexus."

# Standard generation
wav = model.generate(text)
ta.save("test-1.wav", wav, model.sr)

# Voice cloning generation
AUDIO_PROMPT_PATH = "YOUR_FILE.wav"
wav = model.generate(text, audio_prompt_path=AUDIO_PROMPT_PATH)
ta.save("test-2.wav", wav, model.sr)

When calling generate_stream, you can tune the following parameters:

• audio_prompt_path: Path to reference audio for voice cloning.
• chunk_size: Number of speech tokens per chunk (default: 50). Smaller values reduce latency but increase overhead.
• print_metrics: Boolean to enable automatic printing of latency and RTF (default: True).
• exaggeration: Emotion intensity control (range: 0.0 to 1.0+).
• cfg_weight: Classifier-free guidance weight (range: 0.0 to 1.0).
• temperature: Sampling randomness (range: 0.1 to 1.0).

The log_mel_spectrogram method computes the log-Mel spectrogram of an input audio waveform.

Arguments:

• audio: A torch.Tensor or numpy.ndarray containing the audio waveform in 16 kHz.
• padding: (Optional) Number of zero samples to pad to the right. Defaults to 0.

Returns:

• A torch.Tensor of shape (128, n_frames) representing the log-Mel spectrogram.

import torch
import numpy as np

# audio can be a Tensor or NumPy array
audio = torch.randn(16000) 
log_mel = tokenizer.log_mel_spectrogram(audio, padding=0)