DOCSEARCHbyAlgolia
Submit Docs

Qwen3-TTS

repository·main·Indexed 12 days ago

https://github.com/qwenlm/qwen3-tts

A high-performance speech generation series supporting voice cloning, voice design, and natural language-based voice control across 10 languages. It features a discrete multi-codebook LM architecture and a Dual-Track hybrid system for low-latency streaming (as low as 97ms). The series includes models such as Qwen3-TTS-12Hz-1.7B-VoiceDesign, Qwen3-TTS-12Hz-1.7B-CustomVoice, and Qwen3-TTS-12Hz-1.7B-Base, utilizing the Qwen3-TTS-Tokenizer-12Hz for acoustic compression.

Tokens
7.1K
Snippets
19
Records
22
Agent score
93%

What's inside Qwen3-TTS

  1. Overview of Qwen3-TTS capabilities

    main

    Qwen3-TTS is a series of speech generation models supporting 10 languages (Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, and Italian). It features:

    • Powerful Speech Representation: Uses Qwen3-TTS-Tokenizer-12Hz for efficient acoustic compression and high-fidelity reconstruction.
    • Universal End-to-End Architecture: A discrete multi-codebook LM architecture that avoids information bottlenecks found in traditional LM+DiT schemes.
    • Low-Latency Streaming: Supports both streaming and non-streaming generation via a Dual-Track hybrid architecture, with end-to-end latency as low as 97ms.
    • Intelligent Voice Control: Allows control over timbre, emotion, and prosody using natural language instructions.

  2. Workflow: Design a voice then clone it

    main

    You can combine the VoiceDesign and Base models to create a consistent character voice:

    1. Use VoiceDesign model to synthesize a short reference clip matching your target persona using generate_voice_design.
    2. Use the resulting audio and text to build a reusable prompt with create_voice_clone_prompt on the Base model.
    3. Use generate_voice_clone with that prompt to generate many lines of dialogue with the same consistent character voice.
    # 1. Design the voice
design_model = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign", ...)
ref_wavs, sr = design_model.generate_voice_design(text=ref_text, language="English", instruct=ref_instruct)

# 2. Build reusable prompt
clone_model = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-1.7B-Base", ...)
voice_clone_prompt = clone_model.create_voice_clone_prompt(ref_audio=(ref_wavs[0], sr), ref_text=ref_text)

# 3. Generate multiple lines
wavs, sr = clone_model.generate_voice_clone(text=sentences, language="English", voice_clone_prompt=voice_clone_prompt)

  3. Perform offline inference with vLLM-Omni

    main

    Qwen3-TTS is supported by vLLM-Omni for offline inference. You can use the end2end.py script from the vLLM-Omni repository to generate audio output using different task types: CustomVoice, VoiceDesign, or Base.

    To use this, you must first clone the vLLM-Omni repository and navigate to the specific Qwen3-TTS example directory: vllm-omni/examples/offline_inference/qwen3_tts.

    # git clone https://github.com/vllm-project/vllm-omni.git

# cd vllm-omni/examples/offline_inference/qwen3_tts

# Run a single sample with CustomVoice task
python end2end.py --query-type CustomVoice

# Batch sample (multiple prompts in one run) with CustomVoice task:
python end2end.py --query-type CustomVoice --use-batch-sample

# Run a single sample with VoiceDesign task
python end2end.py --query-type VoiceDesign

# Batch sample (multiple prompts in one run) with VoiceDesign task:
python end2end.py --query-type VoiceDesign --use-batch-sample

# Run a single sample with Base task in icl mode-tag
python end2end.py --query-type Base --mode-tag icl

  5. Manually download Qwen3-TTS model weights

    main

    If your environment cannot download weights automatically during execution, you can manually download them using ModelScope (recommended for Mainland China) or Hugging Face.

    # Download through ModelScope (recommended for users in Mainland China)
pip install -U modelscope
modelscope download --model Qwen/Qwen3-TTS-Tokenizer-12Hz  --local_dir ./Qwen3-TTS-Tokenizer-12Hz 
modelscope download --model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --local_dir ./Qwen3-TTS-12Hz-1.7B-CustomVoice
modelscope download --model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign --local_dir ./Qwen3-TTS-12Hz-1.7B-VoiceDesign
modelscope download --model Qwen/Qwen3-TTS-12Hz-1.7B-Base --local_dir ./Qwen3-TTS-12Hz-1.7B-Base
modelscope download --model Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --local_dir ./Qwen3-TTS-12Hz-0.6B-CustomVoice
modelscope download --model Qwen/Qwen3-TTS-12Hz-0.6B-Base --local_dir ./Qwen3-TTS-12Hz-0.6B-Base

# Download through Hugging Face
pip install -U "huggingface_hub[cli]"
huggingface-cli download Qwen/Qwen3-TTS-Tokenizer-12Hz --local-dir ./Qwen3-TTS-Tokenizer-12Hz
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --local-dir ./Qwen3-TTS-12Hz-1.7B-CustomVoice
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign --local-dir ./Qwen3-TTS-12Hz-1.7B-VoiceDesign
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base --local-dir ./Qwen3-TTS-12Hz-1.7B-Base
huggingface-cli download Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --local-dir ./Qwen3-TTS-12Hz-0.6B-CustomVoice
huggingface-cli download Qwen/Qwen3-TTS-12Hz-0.6B-Base --local-dir ./Qwen3-TTS-12Hz-0.6B-Base

  6. Install and set up the Qwen3-TTS environment

    main

    To use Qwen3-TTS, install the qwen-tts package from PyPI in a fresh Python 3.12 environment. It is highly recommended to use FlashAttention 2 to reduce GPU memory usage. FlashAttention 2 requires the model to be loaded in torch.float16 or torch.bfloat16 and compatible hardware.

    Standard Installation:

    conda create -n qwen3-tts python=3.12 -y
conda activate qwen3-tts
pip install -U qwen-tts

    Development Installation (from source):

    git clone https://github.com/QwenLM/Qwen3-TTS.git
cd Qwen3-TTS
pip install -e .

    Install FlashAttention 2:

    pip install -U flash-attn --no-build-isolation

    Note: If your machine has limited RAM (<96GB) but many CPU cores, use MAX_JOBS=4 pip install -U flash-attn --no-build-isolation.

  7. Setup for fine-tuning Qwen3-TTS-12Hz

    main

    To begin fine-tuning the Qwen3-TTS-12Hz-1.7B/0.6B-Base models, you must first install the qwen-tts package and clone the repository to access the fine-tuning scripts.

    Note: The current implementation supports single-speaker fine-tuning only.

    pip install qwen-tts
git clone https://github.com/QwenLM/Qwen3-TTS.git
cd Qwen3-TTS/finetuning

  8. Prepare input JSONL for fine-tuning

    main

    Training data must be provided in a JSONL format (one JSON object per line). Each entry requires the following keys:

    • audio: Path to the target training audio file (.wav).
    • text: The transcript corresponding to the audio.
    • ref_audio: Path to the reference speaker audio file (.wav).

    Best Practice: It is strongly recommended to use the same ref_audio for all samples in the dataset to improve speaker consistency and stability during generation.

    {"audio":"./data/utt0001.wav","text":"其实我真的有发现，我是一个特别善于观察别人情绪的人。","ref_audio":"./data/ref.wav"}
{"audio":"./data/utt0002.wav","text":"She said she would be here by noon.","ref_audio":"./data/ref.wav"}

  9. Launch the Qwen3-TTS Gradio Demo

    main

    Use the qwen-tts-demo command to launch a web-based Gradio interface for interacting with Qwen3 TTS models. The demo interface adapts based on the model type (CustomVoice, VoiceDesign, or Base) loaded from the provided checkpoint.

    Basic Usage

    Provide the model checkpoint path or HuggingFace repository ID as a positional argument:

    qwen-tts-demo Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice

    Common Configuration Flags

    • --device: Specify the device for loading (e.g., cpu, cuda, cuda:0). Defaults to cuda:0.
    • --dtype: Set the torch data type. Options: bfloat16, bf16, float16, fp16, float32, fp32. Defaults to bfloat16.
    • --flash-attn / --no-flash-attn: Enable or disable FlashAttention-2. Defaults to enabled.
    • --ip / --port: Configure the Gradio server bind IP and port. Defaults to 0.0.0.0 and 8000 respectively.
    • --share: Create a public Gradio link. Defaults to disabled.
    • --concurrency: Set the Gradio queue concurrency limit. Defaults to 16.
    # Example: Launching a VoiceDesign model on a specific GPU with float16
qwen-tts-demo Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign --device cuda:0 --dtype float16

# Example: Launching a CustomVoice model with no FlashAttention
qwen-tts-demo Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --no-flash-attn

  10. Test fine-tuned model with generate_custom_voice

    main

    After fine-tuning, you can test the new voice using the Qwen3TTSModel.generate_custom_voice method. Load the model from your checkpoint directory and specify the speaker name used during training.

    import torch
import soundfile as sf
from qwen_tts import Qwen3TTSModel

device = "cuda:0"
tts = Qwen3TTSModel.from_pretrained(
    "output/checkpoint-epoch-2",
    device_map=device,
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)

wavs, sr = tts.generate_custom_voice(
    text="She said she would be here by noon.",
    speaker="speaker_test",
)
sf.write("output.wav", wavs[0], sr)

  11. Use Voice Design with `generate_voice_design`

    main

    For Qwen3-TTS-12Hz-1.7B-VoiceDesign models, use generate_voice_design to create a unique voice based on a natural-language description.

    Parameters:

    • text: The text to synthesize.
    • language: The target language.
    • instruct: A natural-language description of the desired voice characteristics (e.g., age, tone, emotion, pitch).

    This is useful for creating a persona from scratch.

    import torch
import soundfile as sf
from qwen_tts import Qwen3TTSModel

model = Qwen3TTSModel.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign",
    device_map="cuda:0",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)

wavs, sr = model.generate_voice_design(
    text="哥哥，你回来啦，人家等了你好久好久了，要抱抱！",
    language="Chinese",
    instruct="体现撒娇稚嫩的萝莉女声，音调偏高且起伏明显，营造出黏人、做作又刻意卖萌的听觉效果",
)
sf.write("output_voice_design.wav", wavs[0], sr)

  12. Use Voice Cloning with `generate_voice_clone`

    main

    For Qwen3-TTS-12Hz-1.7B/0.6B-Base models, use generate_voice_clone to clone a voice from a reference audio clip.

    Parameters:

    • text: The text to synthesize.
    • language: The target language.
    • ref_audio: The reference audio. Can be a local file path, a URL, a base64 string, or a (numpy_array, sample_rate) tuple.
    • ref_text: The transcript of the ref_audio.
    • x_vector_only_mode (bool): If True, only the speaker embedding is used; ref_text is not required, but quality may decrease.
    • voice_clone_prompt: A pre-computed prompt object (created via create_voice_clone_prompt) to avoid re-extracting features for multiple generations.

    Optimization: To reuse a voice across multiple calls, use model.create_voice_clone_prompt(ref_audio, ref_text) once and pass the result to voice_clone_prompt in subsequent generate_voice_clone calls.

    import torch
import soundfile as sf
from qwen_tts import Qwen3TTSModel

model = Qwen3TTSModel.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-1.7B-Base",
    device_map="cuda:0",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)

ref_audio = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-TTS-Repo/clone.wav"
ref_text  = "Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you."

wavs, sr = model.generate_voice_clone(
    text="I am solving the equation: x = [-b ± √(b²-4ac)] / 2a? Nobody can — it's a disaster (◍•͈⌔•͈◍), very sad!",
    language="English",
    ref_audio=ref_audio,
    ref_text=ref_text,
)
sf.write("output_voice_clone.wav", wavs[0], sr)