CosyVoice2-Ex

CosyVoice2-Ex is an extended version of CosyVoice providing advanced Text-to-Speech (TTS) capabilities. Key features include:

• Pre-trained voices
• 3s rapid voice cloning
• Natural language control
• Automatic speech recognition
• Voice saving functionality
• API support

It is compatible with Windows, Linux, and macOS.

Deployment on Linux and macOS requires a Conda environment. Follow these steps:

1. Create and activate a Conda environment with Python 3.10.
2. Install pynini via conda-forge.
3. Install dependencies via pip using the specified Aliyun mirror.
4. Run the WebUI.

Note: Deployment has been verified on MacBook Pro (M4 Pro) and WSL2 Ubuntu 22.04.

conda create -n cosyvoice -y python=3.10
conda activate cosyvoice
conda install -y -c conda-forge pynini==2.1.5
pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com

python webui.py --port 8080 --open

On Linux, you can optionally install ttsfrd to enhance text normalization performance. This requires installing system dependencies, cloning the CosyVoice-ttsfrd repository into pretrained_models, and installing the provided .whl file.

# Install system dependencies
sudo apt install -y git build-essential curl wget ffmpeg unzip git git-lfs sox libsox-dev nvidia-cuda-toolkit

# Setup ttsfrd
git lfs install
git clone https://www.modelscope.cn/iic/CosyVoice-ttsfrd.git pretrained_models/CosyVoice-ttsfrd

cd pretrained_models/CosyVoice-ttsfrd/
unzip resource.zip -d .
pip install ttsfrd-0.3.6-cp38-cp38-linux_x86_64.whl

For Windows users, a one-click package is available.

1. Download the latest release from the GitHub releases page.
2. Extract the package.
3. Double-click 运行-CosyVoice2-Ex.bat to start the application.

Note: The first run will automatically download approximately 10GB of model files. Alternatively, you can manually download the models from Modelscope and place them in the pretrained_models directory.

Download from: https://github.com/journey-ad/CosyVoice2-Ex/releases/latest

The CosyVoice server provides inference endpoints for various TTS modes (SFT, Zero-shot, Cross-lingual, and Instruct). You can start the server using the Python script with the following CLI arguments:

• --port: The port number to host the server on (default: 50000).
• --model_dir: The local path to the model or a ModelScope repository ID (default: iic/CosyVoice-300M).

The server automatically attempts to load the model using CosyVoice first, and if that fails, it falls back to CosyVoice2.

python server.py --port 50000 --model_dir /path/to/your/model

The matcha module is part of the Matcha-TTS third-party dependency. If you encounter this error, it means the submodule is missing or not in your Python path.

1. Initialize submodules: If the third_party/Matcha-TTS directory is empty, run:

   git submodule update --init --recursive

2. Update PYTHONPATH: To use from cosyvoice.cli.cosyvoice import CosyVoice within a Python script, you must add the submodule to your environment's search path:

   export PYTHONPATH=third_party/Matcha-TTS

These errors typically occur when Git LFS is missing or the ttsfrd resources are not correctly extracted. Follow these steps to properly install the ttsfrd dependency:

1. Ensure git-lfs is installed on your system.
2. Clone the required model repository into the pretrained_models/CosyVoice-ttsfrd directory.
3. Unzip the resource.zip file into the current directory.
4. Install the provided wheel file via pip.

git clone https://www.modelscope.cn/iic/CosyVoice-ttsfrd.git pretrained_models/CosyVoice-ttsfrd
cd pretrained_models/CosyVoice-ttsfrd/
unzip resource.zip -d .
pip install ttsfrd-0.3.6-cp38-cp38-linux_x86_64.whl

To use the API, first start the API server by running python api.py. The server defaults to http://localhost:9880/.

Query Parameters

• text: The text you want to synthesize.
• speaker: The name of the target speaker/voice.
• instruct (optional): Natural language instructions to control the speech style (e.g., speed, tone).

# Start the API server
python api.py

# Example 1: Basic text and speaker
http://localhost:9880/?text=春日清晨，老街深处飘来阵阵豆香。&speaker=舌尖上的中国

# Example 2: Text with natural language instructions
http://localhost:9880/?text=hello%20hello~%20[breath]%20听得到吗？&speaker=嘉然&instruct=慢速，用可爱的语气说

The vc method performs voice conversion using provided speech tokens.

Method Signature: vc(self, source_speech_token, flow_prompt_speech_token, prompt_speech_feat, flow_embedding, stream=False, speed=1.0, **kwargs)

Arguments:

• source_speech_token: The speech tokens to be converted.
• flow_prompt_speech_token: Speech tokens for the flow prompt.
• prompt_speech_feat: Mel-spectrogram features for the prompt.
• flow_embedding: Embedding for the flow model.
• stream: If True, returns a generator yielding chunks of speech.
• speed: (Optional) Speed factor (only supported in non-stream mode).

Returns:

• A generator yielding dictionaries in the format {'tts_speech': torch.Tensor}.

# Voice conversion usage
for chunk in model.vc(source_token, flow_prompt_token, prompt_feat, flow_emb):
    audio = chunk['tts_speech']

Use the load method to populate the initialized model modules with state dictionaries from saved files.

Method Signature: load(self, llm_model, flow_model, hift_model)

Arguments:

• llm_model (str): Path to the LLM state dict file.
• flow_model (str): Path to the flow state dict file.
• hift_model (str): Path to the HIFT state dict file. Note: If the model is a HiFi-GAN model, the method automatically handles the 'generator.' prefix in keys.

model.load(
    llm_model='path/to/llm.pth',
    flow_model='path/to/flow.pth',
    hift_model='path/to/hift.pth'
)

Use frontend_vc to prepare inputs for voice conversion. This mode transforms a source speech into the style of a prompt speech.

Parameters:

• source_speech_16k: The 16kHz audio tensor of the speech you want to convert.
• prompt_speech_16k: The 16kHz audio tensor of the target voice style.
• resample_rate: The target sampling rate.

model_input = frontend.frontend_vc(
    source_speech_16k=source_audio,
    prompt_speech_16k=target_style_audio,
    resample_rate=24000
)