Chatterbox: Open Source TTS and Voice Conversion by Resemble AI

Use the following parameter adjustments to fine-tune the output of Chatterbox TTS:

General Use (TTS and Voice Agents)

• Default Settings: exaggeration=0.5 and cfg_weight=0.5 work well for most prompts.
• Language Consistency: If the reference clip's language differs from the target language tag, set cfg_weight to 0 to prevent the output from inheriting the reference clip's accent.
• Fast Speakers: If the reference speaker has a fast style, lower cfg_weight to approximately 0.3 to improve pacing.

Expressive or Dramatic Speech

• Dramatic Effect: Increase exaggeration to 0.7 or higher.
• Pacing Control: Higher exaggeration tends to speed up speech. To maintain deliberate, slower pacing while using high exaggeration, lower cfg_weight to approximately 0.3.

To install from source, create a Python 3.11 environment, clone the repository, and install in editable mode. This is recommended if you need to modify the code or dependencies.

# conda create -yn chatterbox python=3.11
# conda activate chatterbox

git clone https://github.com/resemble-ai/chatterbox.git
cd chatterbox
pip install -e .

You can install the Chatterbox TTS package via pip. For development or if you need to modify the source, you can install it from source using a Conda environment with Python 3.11.

pip install chatterbox-tts

Every audio file generated by Chatterbox includes Resemble AI's Perth (Perceptual Threshold) Watermarker. These are imperceptible neural watermarks designed to survive MP3 compression and audio editing. You can extract the watermark using the perth library and librosa. The extraction returns 1.0 if a watermark is present or 0.0 if no watermark is detected.

import perth
import librosa

AUDIO_PATH = "YOUR_FILE.wav"

# Load the watermarked audio
watermarked_audio, sr = librosa.load(AUDIO_PATH, sr=None)

# Initialize watermarker (same as used for embedding)
watermarker = perth.PerthImplicitWatermarker()

# Extract watermark
watermark = watermarker.get_watermark(watermarked_audio, sample_rate=sr)
print(f"Extracted watermark: {watermark}")
# Output: 0.0 (no watermark) or 1.0 (watermarked)

The ChatterboxNano model (110M parameters) shares the same architecture as Turbo but is designed for on-device or CPU inference. It can be loaded using the ChatterboxTurboTTS class by setting nano=True. It also supports paralinguistic tags.

import torchaudio as ta
import torch
from chatterbox.tts_turbo import ChatterboxTurboTTS

# Load the Nano model (also runs on CPU: device="cpu")
model = ChatterboxTurboTTS.from_pretrained(device="cuda", nano=True)

# Generate with Paralinguistic Tags
text = "Hi there, Sarah here from MochaFone calling you back [chuckle], have you got one minute to chat about the billing issue?"

# Generate audio (requires a reference clip for voice cloning)
wav = model.generate(text, audio_prompt_path="your_10s_ref_clip.wav")

ta.save("test-nano.wav", wav, model.sr)

The ChatterboxTurboTTS class provides access to the Turbo model (350M parameters), which is optimized for low-latency English voice agents. It supports native paralinguistic tags like [laugh] and [chuckle]. Use audio_prompt_path in the generate method to perform zero-shot voice cloning with a reference clip.

import torchaudio as ta
import torch
from chatterbox.tts_turbo import ChatterboxTurboTTS

# Load the Turbo model
model = ChatterboxTurboTTS.from_pretrained(device="cuda")

# Generate with Paralinguistic Tags
text = "Hi there, Sarah here from MochaFone calling you back [chuckle], have you got one minute to chat about the billing issue?"

# Generate audio (requires a reference clip for voice cloning)
wav = model.generate(text, audio_prompt_path="your_10s_ref_clip.wav")

ta.save("test-turbo.wav", wav, model.sr)

The general-purpose Chatterbox Multilingual model supports the following 23 languages via their ISO codes:

• Arabic (ar)
• Danish (da)
• German (de)
• Greek (el)
• English (en)
• Spanish (es)
• Finnish (fi)
• French (fr)
• Hebrew (he)
• Hindi (hi)
• Italian (it)
• Japanese (ja)
• Korean (ko)
• Malay (ms)
• Dutch (nl)
• Norwegian (no)
• Polish (pl)
• Portuguese (pt)
• Russian (ru)
• Swedish (sv)
• Swahili (sw)
• Turkish (tr)
• Chinese (zh)

The EnTokenizer class is used for English-specific tokenization. It requires a path to a vocabulary file. It automatically validates that the vocabulary contains the required special tokens [START] and [STOP].

Key methods:

• encode(txt: str): Cleans text by replacing spaces with the [SPACE] token and returns token IDs.
• decode(seq): Converts token sequences (including torch.Tensor) back into a cleaned string, handling special tokens like [SPACE], [STOP], and [UNK].

from chatterbox.models.tokenizers.tokenizer import EnTokenizer

tokenizer = EnTokenizer("path/to/vocab.json")
ids = tokenizer.encode("hello world")
text = tokenizer.decode(torch.tensor(ids))

Because the S3 tokenizer operates at a fixed rate of 25 tokens per second, input audio waveforms should be padded so their length is a multiple of 40ms. The pad method automates this process.

Arguments:

• wavs: A list of audio waveforms (as np.ndarray or torch.Tensor).
• sr: The sampling rate of the input audio (should be 16,000 for S3).

Returns: A list of torch.Tensor objects, each padded to the required length.

import numpy as np

# Example: padding a single numpy array
# Note: sr should be 16000 for S3
padded_wavs = tok.pad([my_numpy_audio], sr=16000)
processed_wav = padded_wavs[0]

The log_mel_spectrogram method computes the log-Mel spectrogram of an audio waveform.

Arguments:

• audio: A torch.Tensor or np.ndarray containing the 16 kHz audio waveform.
• padding: (Optional) Number of zero samples to pad to the right. Defaults to 0.

Returns: A torch.Tensor of shape (128, n_frames) containing the processed log-Mel spectrogram.

import torch

# audio_tensor must be 16kHz
mel_spec = tok.log_mel_spectrogram(audio_tensor, padding=0)

Chatterbox provides several standalone normalization functions for specific languages. Note that some require optional dependencies to be installed:

• Japanese: hiragana_normalize(text: str) converts Kanji to Hiragana while preserving Katakana. Requires pykakasi.
• Hebrew: add_hebrew_diacritics(text: str) adds diacritics to Hebrew text. Requires dicta_onnx.
• Korean: korean_normalize(text: str) decomposes Hangul syllables into Jamo components.
• Russian: add_russian_stress(text: str) adds stress marks to Russian text. Requires russian_text_stresser.
• Chinese: ChineseCangjieConverter converts Chinese characters to Cangjie codes. It downloads mapping files from the ResembleAI/chatterbox HuggingFace repository and uses pkuseg for segmentation if available.