faster-qwen3-tts

FasterQwen3TTS uses StaticCache for maximum throughput, while the upstream Qwen3-TTS uses DynamicCache. While the underlying algorithms are mathematically equivalent, they differ in execution:

• Static Cache (FasterQwen3TTS): Uses fixed max-length KV buffers and an explicit attention mask. This typically selects a different SDPA (Scaled Dot Product Attention) kernel (e.g., masked attention).
• Dynamic Cache (Upstream): Uses the current sequence length and can use is_causal=True without an explicit mask, typically selecting a different kernel.

Note on Precision: Because different kernels and reduction orders are used, outputs in BF16/TF32 are not bit-exact and may differ slightly between the two implementations.

FasterQwen3TTS achieves significantly higher Real-Time Factor (RTF) and lower Time to First Audio (TTFA) compared to the standard Qwen3-TTS reference implementation by using PyTorch CUDA Graphs.

This optimization eliminates kernel launch overhead—the time the GPU spends idling while waiting for the CPU to dispatch the ~500 small GPU operations required for each decode step. This is particularly effective on hardware where there is a CPU/GPU imbalance (e.g., fast GPUs with slower CPUs like the Jetson AGX Orin, or high-end consumer GPUs like the RTX 4090).

Key Performance Metrics (RTX 4090):

• 0.6B Model: ~5.53 RTF with ~154ms TTFA.
• 1.7B Model: ~4.78 RTF with ~171ms TTFA.

Key Performance Metrics (Jetson AGX Orin):

• 0.6B Model: ~1.57 RTF with ~556ms TTFA (a 9.0x speedup over baseline).

Faster Qwen3-TTS achieves high performance by using CUDA graphs to capture the entire decode step of two autoregressive transformers: the Talker (28 layers) and the Code Predictor (5 layers).

Instead of launching ~500 small CUDA kernels per step with high Python overhead, the system uses torch.cuda.CUDAGraph to replay the entire step as a single GPU operation. This is achieved through:

1. Static KV cache: Pre-allocated fixed-size tensors to avoid dynamic allocation.
2. Model's own forward: Utilizing SDPA + RoPE via native attention layers.
3. Graph capture: Capturing both the predictor and talker.
4. Padded attention: Using attention masks to handle variable-length KV within the fixed buffers.

The non_streaming_mode parameter controls how text is fed into the generation process (step-by-step vs. full utterance). The API uses None as a sentinel to preserve upstream defaults:

• generate_voice_clone and generate_voice_clone_streaming: None resolves to False (step-by-step text feeding).
• generate_custom_voice, generate_custom_voice_streaming, generate_voice_design, and generate_voice_design_streaming: None resolves to True (full utterance preparation).

Note for GGML backend: Passing non_streaming_mode=False currently emits a warning because the GGML backend does not yet support the step-by-step text feeding ABI switch; it will use its native prompt layout instead.

Faster Qwen3-TTS supports streaming output where audio chunks are yielded during generation using CUDA graphs. The streaming generator yields codec ID chunks every chunk_size steps. To prevent boundary artifacts, the model wrapper decodes each chunk using a sliding window with a 25-frame left context.

In Python, the streaming methods are pull-based generators. For real-time local playback, it is recommended to use a queue-backed player like StreamPlayer. Note that blocking after each yielded chunk will prevent generation and playback from overlapping.

The project supports streaming output, which yields audio chunks during generation to minimize Time-To-First-Audio (TTFA). This is achieved by accumulating codec tokens in configurable chunk sizes and decoding them using the same CUDA graphs used for non-streaming generation.

On hardware like the Jetson AGX Orin (0.6B model), the choice of chunk_size significantly impacts performance:

• Small chunk_size (e.g., 1 or 2): Lower TTFA and better real-time performance (RTF), but smaller audio chunks.
• Large chunk_size (e.g., 12): Higher TTFA and higher RTF, but larger audio chunks (e.g., 1000ms).

For real-time applications on constrained hardware, a chunk_size=2 is recommended as the smallest size that maintains real-time performance. On high-end GPUs, chunk_size=1 is typically sufficient to stay above RTF 1.0.

To set up the project, clone the repository, run the setup script to create a virtual environment (using uv) and download models, and then run the benchmark script to evaluate streaming performance.

git clone https://github.com/andimarafioti/faster-qwen3-tts
cd faster-qwen3-tts
./setup.sh       # creates venv with uv, installs deps, downloads models
./benchmark.sh   # runs streaming benchmark, saves JSON + audio samples

To compare the performance of non-streaming mode (non_streaming_mode=True) against streaming mode (non_streaming_mode=False) during In-Context Learning (ICL) voice cloning, use the following configuration parameters:

• ICL Configuration: Set xvec_only=False to ensure the reference audio is used in the context.
• Model: Qwen/Qwen3-TTS-12Hz-1.7B-Base.
• Generation Settings:
  • max_new_tokens=168
  • temperature=0.9
  • top_k=50
  • top_p=1.0
  • do_sample=True
  • language="English"
• Reproducibility: Use a consistent seed calculated as 1337 + ref_index*10 + prompt_index for both modes to ensure a fair comparison.

Generated files follow the pattern: icl_<ref_key>_gen<1|2>_nsm_<true|false>.wav.

If your NVIDIA driver is older (e.g., CUDA 12.4 hosts like T4, A10G, AWS, Azure ML), install the specific PyTorch wheel matching your driver version to avoid initialization errors.

pip install "torch==2.5.1" "torchaudio==2.5.1" --index-url https://download.pytorch.org/whl/cu124

Since the static and dynamic cache paths use different kernels, the audio outputs are not bit-identical. To evaluate the perceptual impact of these optimizations, you can compare side-by-side samples provided in the repository.

Comparison samples include:

• CustomVoice (predefined speaker IDs)
• ICL (In-Context Learning) (voice-cloning via reference audio)

Audio files and sample indices are located in: samples/parity/.

If you prefer not to use the batch scripts, follow these steps to set up the environment manually:

1. Create a virtual environment:

   python -m venv .venv

2. Activate the environment:

   .venv\Scripts\activate

3. Install dependencies:

   pip install --upgrade pip
   pip install -e .

4. Install flash-attn (optional) (requires a compiler):

   pip install flash-attn

5. Download models using huggingface_hub:

   python -c "from huggingface_hub import snapshot_download; [snapshot_download(f'Qwen/{m}') for m in ['Qwen3-TTS-12Hz-0.6B-Base', 'Qwen3-TTS-12Hz-1.7B-Base']]"