ZipVoice Documentation

The run.sh script automates the deployment workflow through 8 distinct stages. You can execute a range of stages by providing a start stage, an end stage, and an optional model name (zipvoice or zipvoice_distill).

Usage:

bash run.sh <start_stage> <stop_stage> [model_name]

Available Stages:

• Stage 1: Downloads ZipVoice models from HuggingFace
• Stage 2: Exports models to TensorRT format and builds optimized engines
• Stage 3: Creates Triton model repository and configuration files
• Stage 4: Launches Triton Inference Server
• Stage 5: Runs gRPC benchmark tests with multiple concurrency levels
• Stage 6: Tests HTTP client with sample audio
• Stage 7: Launches PyTriton server with speaker caching
• Stage 8: Tests PyTriton server with speaker cache benchmarks

Common Tasks:

• Complete setup and launch Triton (Stages 1-4):

  bash run.sh 1 4 zipvoice_distill

• Launch PyTriton server with speaker caching (Stage 7):

  bash run.sh 7 7 zipvoice_distill

# Complete setup and launch (stages 1-4)
bash run.sh 1 4 zipvoice_distill

Use the zipvoice.bin.infer_zipvoice module to generate speech for a single speaker. You can perform inference on a single sentence or a list of sentences provided in a .tsv file.

Single sentence inference

Provide a prompt waveform, its transcription, and the target text to be synthesized.

Batch inference

Provide a .tsv file where each line follows the format: {wav_name}\t{prompt_transcription}\t{prompt_wav}\t{text}.

Model variants

• zipvoice: Standard model for high quality.
• zipvoice_distill: Distilled model for faster inference.

Special Tokens

• <>: Used for Chinese pinyin (e.g., <chang2>).
• []: Used for other special tags.

# Inference of a single sentence
python3 -m zipvoice.bin.infer_zipvoice \
    --model-name zipvoice \
    --prompt-wav prompt.wav \
    --prompt-text "I am the transcription of the prompt wav." \
    --text "I am the text to be synthesized." \
    --res-wav-path result.wav

# Inference of a list of sentences
python3 -m zipvoice.bin.infer_zipvoice \
    --model-name zipvoice \
    --test-list test.tsv \
    --res-dir results

Depending on your hardware and priority (quality vs. speed), use the following optimization strategies:

For Single-Speaker Models

• Use Distilled Models: Switch --model-name to zipvoice_distill.
• Reduce Steps: For distilled models, set --num-steps as low as 4 (default is 8).
• ONNX (CPU only): Use zipvoice.bin.infer_zipvoice_onnx. Use --onnx-int8 True for INT8 quantization (note: this may degrade quality).
• TensorRT (NVIDIA GPU): Export the model using zipvoice.bin.tensorrt_export for ~2x throughput compared to PyTorch.

For CPU Inference

• Multi-threading: Use the --num-thread parameter (e.g., --num-thread 4). Default is 1.

General Tips

• Prompt Length: Keep prompt wavs short (<3s for single-speaker, <10s for dialogue) to prevent slow inference and quality degradation.
• Avoid ONNX on GPU: It is slower than PyTorch on GPU hardware.

Use the zipvoice.bin.infer_zipvoice_dialog module to generate two-party spoken dialogues.

Model variants

• zipvoice_dialog: Generates mono dialogues.
• zipvoice_dialog_stereo: Generates stereo dialogues.

Input Formats (TSV)

Each line in the --test-list file must follow one of these two formats:

1. Merged prompt format: One prompt wav containing both speakers. Format: {wav_name}\t{prompt_transcription}\t{prompt_wav}\t{text} Example prompt_transcription: "[S1] Hello. [S2] How are you?" Example text: "[S1] I'm fine. [S2] What's your name? [S1] I'm Eric. [S2] Hi Eric."

2. Splitted prompt format: Separate prompt wavs for each speaker. Format: {wav_name}\t{spk1_prompt_transcription}\t{spk2_prompt_transcription}\t{spk1_prompt_wav}\t{spk2_prompt_wav}\t{text}

python3 -m zipvoice.bin.infer_zipvoice_dialog \
    --model-name "zipvoice_dialog" \
    --test-list test.tsv \
    --res-dir results