LuxTTS Documentation

To achieve the best results with LuxTTS:

• Use a reference audio file that is at least 3 seconds long.
• If you hear metallic sounds in the output, set return_smooth = True.
• To balance pronunciation accuracy vs. quality, adjust t_shift: lower values reduce pronunciation errors but may lower quality.

To install LuxTTS locally, clone the repository and install the required dependencies using pip.

git clone https://github.com/ysharma3501/LuxTTS.git
cd LuxTTS
pip install -r requirements.txt

The prepare_dataset.py script generates Lhotse manifest files (.jsonl.gz) from TSV files for custom dataset training.

TSV File Formats

Each line in your TSV file must follow one of these two formats:

1. Full audio: {uniq_id}\t{text}\t{wav_path}
2. Partial audio: {uniq_id}\t{text}\t{wav_path}\t{start_time}\t{end_time} (where times are in seconds).

Note: {uniq_id} must be unique for every line.

Usage Example

To prepare a training subset:

python3 -m zipvoice.bin.prepare_dataset \
    --tsv-path data/raw/custom_train.tsv \
    --prefix "custom" \
    --subset "train" \
    --num-jobs 20 \
    --output-dir "data/manifests"

This will produce data/manifests/custom_cuts_train.jsonl.gz.

The training script automatically manages checkpoints in the --exp-dir:

• Periodic Saving: Checkpoints are saved every --save-every-n batches as checkpoint-{batch_idx}.pt.
• Epoch Saving: Checkpoints are saved at the end of each epoch as epoch-{epoch_number}.pt.
• Best Models: The script saves best-train-loss.pt and best-valid-loss.pt when a new best loss is achieved.
• Resuming: To resume training from a specific epoch, use --start-epoch {N}. The script will attempt to load exp-dir/epoch-{N-1}.pt.
• Retention: Use --keep-last-k to limit the number of periodic checkpoints kept on disk to prevent storage exhaustion.

Use the infer_zipvoice_dialog.py CLI tool to perform inference on dialog-style audio. The script supports both mono (zipvoice_dialog) and stereo (zipvoice_dialog_stereo) models. It can load models from a local directory or automatically download pretrained models from Hugging Face.

To run inference, you must provide a --test-list file containing the prompts and target text. The script will iterate through the list, generate the speech, and save the resulting .wav files to the specified --res-dir.

Use the zipvoice.bin.train_zipvoice_dialog module to train a ZipVoice-Dialog model on either the opendialog dataset or a custom dataset. The script supports Distributed Data Parallel (DDP) training across multiple GPUs, mixed-precision training with FP16, and checkpoint resuming.

Basic Usage Example

python3 -m zipvoice.bin.train_zipvoice_dialog \
    --world-size 8 \
    --use-fp16 1 \
    --base-lr 0.0001 \
    --max-duration 500 \
    --checkpoint download/zipvoice/model.pt \
    --model-config conf/zipvoice_base.json \
    --token-file "data/tokens_dialog.txt" \
    --dataset opendialog \
    --manifest-dir data/fbank \
    --exp-dir exp/zipvoice_dialog

Use the zipvoice.bin.tensorrt_export module to export pre-trained ZipVoice or ZipVoice-Distill models from PyTorch to ONNX and subsequently to a TensorRT engine. This is useful for optimizing inference performance on NVIDIA hardware.

Requirements:

• A model directory containing the checkpoint (.pt or .safetensors), model.json, and tokens.txt.
• NVIDIA TensorRT installed in the environment.

Available Model Types:

• zipvoice: The standard model.
• zipvoice_distill: The distilled version of the model.

python3 -m zipvoice.bin.tensorrt_export \
    --model-name zipvoice_distill \
    --model-dir models/zipvoice_distill \
    --checkpoint-name model.pt \
    --trt-engine-file-name fm_decoder.fp16.max_batch_4.plan \
    --tensorrt-model-dir models/zipvoice_distill_trt || exit 1

Generate speech from a single text prompt using a reference audio file. You must provide the reference audio (--prompt-wav), its transcription (--prompt-text), and the target text (--text).

If no local model directory is specified via --model-dir, the script will automatically download the required pre-trained checkpoints from HuggingFace.

python3 -m zipvoice.bin.infer_zipvoice_onnx \
    --onnx-int8 False \
    --model-name zipvoice \
    --prompt-wav prompt.wav \
    --prompt-text "I am a prompt." \
    --text "I am a sentence." \
    --res-wav-path result.wav

Generate speech for multiple sentences provided in a TSV file. Each line in the --test-list file must follow the format: {wav_name}\t{prompt_transcription}\t{prompt_wav}\t{text}.

Results will be saved to the directory specified by --res-dir (defaults to results).

python3 -m zipvoice.bin.infer_zipvoice_onnx \
    --onnx-int8 False \
    --model-name zipvoice \
    --test-list test.tsv \
    --res-dir results

To generate speech, first encode a reference audio file (wav or mp3) using encode_prompt, then pass the text and encoded prompt to generate_speech.

import soundfile as sf
from IPython.display import Audio

text = "Hey, what's up? I'm feeling really great if you ask me honestly!"
prompt_audio = 'audio_file.wav'

# encode audio (takes ~10s to init librosa first time)
encoded_prompt = lux_tts.encode_prompt(prompt_audio, rms=0.01)

# generate speech
final_wav = lux_tts.generate_speech(text, encoded_prompt, num_steps=4)

# save audio
final_wav = final_wav.numpy().squeeze()
sf.write('output.wav', final_wav, 48000)

Import LuxTTS from zipvoice.luxvoice to load the model. You can specify the device to use: cuda for GPU, cpu for CPU (with optional threads parameter), or mps for Apple Silicon Macs.

from zipvoice.luxvoice import LuxTTS

# load model on GPU
lux_tts = LuxTTS('YatharthS/LuxTTS', device='cuda')

# load model on CPU
# lux_tts = LuxTTS('YatharthS/LuxTTS', device='cpu', threads=2)

# load model on MPS for macs
# lux_tts = LuxTTS('YatharthS/LuxTTS', device='mps')