CosyVoice

To prevent the LLM and Token2Wav components from competing for GPU resources, use a Disaggregated Server configuration. In this setup:

• The LLM is deployed on a dedicated GPU (e.g., GPU 0) to leverage in-flight batching.
• The Token2Wav module is deployed on separate, dedicated GPUs (e.g., GPUs 1-3).

This configuration can be launched using Stage 7 of the orchestration script, and benchmarked using Stage 8.

Select one of the following three solutions based on your model requirements:

1. CosyVoice3: Designed for Fun-CosyVoice3-0.5B-2512. The pipeline includes audio_tokenizer, speaker_embedding, token2wav, and vocoder modules managed by Triton, with the LLM served via trtllm-serve.
2. CosyVoice2 + UNet Token2Wav: The baseline acceleration solution for CosyVoice2, utilizing the original UNet-based flow-matching Token2Wav module.
3. CosyVoice2 + DiT Token2Wav: Replaces the UNet Token2Wav with a DiT-based module (from Step-Audio2). This solution supports disaggregated deployment, allowing the LLM and Token2Wav to run on separate GPUs to optimize resource utilization during high concurrency.

CosyVoice supports two primary modes of operation. The configuration is handled during Stage 2 of the run.sh workflow via the Decoupled parameter:

• Streaming TTS: Set Decoupled=True. This is optimized for low-latency chunked delivery.
• Offline TTS: Set Decoupled=False. This is used for full-sentence generation.

Important: If you switch between these modes, you must rerun run.sh starting from Stage 2 to reconfigure the Triton model repository.

The run_cosyvoice3.sh script manages the lifecycle of the CosyVoice3 deployment through numbered stages. You can execute specific ranges of stages using the syntax: bash run_cosyvoice3.sh <start_stage> <stop_stage>.

# Example: Run stages 0 through 3 to prepare models and launch the server
bash run_cosyvoice3.sh 0 3

To fine-tune CosyVoice2 using GRPO, it is recommended to use the pre-built Docker image containing the necessary veRL and vLLM dependencies. If Docker is unavailable, you can install dependencies locally by following the instructions in run.sh stage -2.

docker pull soar97/verl:app-verl0.4-vllm0.8.5-mcore0.12.2-te2.2

After training, follow these steps to evaluate and use your model:

1. Merge and Export Checkpoint (Stage 3): Collect sharded FSDP weights and merge them into a Hugging Face-style checkpoint.

   bash run.sh 3 3
   # Merges weights into $llm_path/merged_hf_model

2. Evaluate (Stage 4): Run distributed inference via infer_dataset.py to compute Word Error Rate (WER). You can evaluate on the CosyVoice3 zero-shot Chinese set or the Seed-TTS test set (by setting dataset=test_zh).

   bash run.sh 4 4

3. Convert for CosyVoice Repository (Stage 5): Convert the Hugging Face checkpoint back into the format expected by the official CosyVoice repository.

   bash run.sh 5 5

   Note: A slight accuracy drop may occur when using the converted model compared to the direct Hugging Face format.

Each acceleration solution can be launched using a specific Docker Compose configuration file. Run the command corresponding to your chosen solution to start the environment.

# CosyVoice3
docker compose -f docker-compose.cosyvoice3.yml up

# CosyVoice2 + UNet Token2Wav
docker compose -f docker-compose.cosyvoice2.unet.yml up

# CosyVoice2 + DiT Token2Wav
docker compose -f docker-compose.cosyvoice2.dit.yml up

Run the GRPO training process using stage 2 of the run.sh script. This invokes verl.trainer.main_ppo with specific configurations.

Key Arguments:

• algorithm.adv_estimator=grpo: Switches the algorithm from PPO to GRPO.
• data.train_files: Path to the training parquet file.
• data.val_files: Path to the validation parquet file.
• custom_reward_function.path: Path to the reward script (e.g., reward_tts.py).

Note: The lm_head bias is disabled during training to ensure compatibility with VLLM and Transformers' Qwen models.

bash run.sh 2 2

The GRPO training requires a reward server to calculate scores based on speech synthesis quality. The server performs the following:

1. Converts generated speech tokens to a 16 kHz waveform using the CosyVoice2 pretrained U-Net.
2. Transcribes the waveform using SenseVoice ASR.
3. Calculates the pinyin-level error rate against the ground-truth text (score 0-1).

Start the server in a dedicated terminal or on a separate GPU using stage 1. The Triton server will listen on ports 8000, 8001, and 8002. The custom reward implementation in reward_tts.py communicates with this server.

bash run.sh 1 1

Using TensorRT-LLM can provide up to 4x acceleration for CosyVoice2 LLM inference compared to the HuggingFace Transformers implementation. This is managed via Docker Compose in the runtime/triton_trtllm directory.

cd runtime/triton_trtllm
docker compose up -d