VoxCPM: Tokenizer-Free TTS

VoxCPM2 is a tokenizer-free speech synthesis system based on a diffusion autoregressive architecture. It generates continuous speech representations directly, bypassing discrete audio encoding steps to achieve highly natural and expressive speech.

Built on the MiniCPM-4 base with 2 billion parameters, it was trained on over 2 million hours of multilingual audio.

Key Capabilities:

• Multilingual Synthesis: Supports 30 global languages and 9 Chinese dialects via raw text input.
• Voice Design: Create entirely new voices using natural language descriptions (e.g., gender, age, emotion, speed) without needing reference audio.
• Controllable Voice Cloning: Clone voices from reference audio clips while applying style instructions to control emotion, speed, and expressiveness.
• Ultimate Cloning: Provide both reference audio and its transcript for seamless continuation and precise detail reproduction.
• High-Quality Output: Native 48kHz audio output (using AudioVAE V2) even when input reference audio is 16kHz.
• Real-time Streaming: Achieves low Real-Time Factor (RTF) on NVIDIA RTX 4090 hardware, especially when accelerated via Nano-vLLM or vLLM-Omni.

VoxCPM2 is a tokenizer-free TTS model that achieves state-of-the-art or comparable results across several public benchmarks:

• Seed-TTS-eval: High performance in zero-shot and controllable TTS, with competitive WER (Word Error Rate) and SIM (Similarity) scores in English and Chinese.
• CV3-eval: Strong multilingual performance across various languages including Chinese, English, Japanese, Korean, German, Spanish, French, Italian, and Russian.
• MiniMax-Multilingual-Test: Demonstrates high similarity (SIM) and low WER across a wide array of languages, including Arabic, Cantonese, Dutch, Finnish, Greek, Hindi, and more.
• Internal 30-Language ASR Benchmark: Shows high intelligibility with an average CER/WER of 1.68% across 30 languages (evaluated via Gemini 3.1 Flash Lite API).
• InstructTTSEval: High scores in instruction-guided voice design, particularly in English (APS, DSD, and RP metrics).

VoxCPM supports two fine-tuning methods to adapt the model to specific speakers, languages, or domains using as little as 5–10 minutes of audio:

1. LoRA fine-tuning: A parameter-efficient method recommended for most users.
2. Full fine-tuning (SFT): Updates all model parameters.

You can run training via CLI scripts or use the provided WebUI for a graphical interface.

For detailed instructions on data preparation, configuration, and LoRA hot-swapping, refer to the Fine-tuning Guide.

# LoRA fine-tuning (parameter-efficient, recommended)
python scripts/train_voxcpm_finetune.py \
    --config_path conf/voxcpm_v2/voxcpm_finetune_lora.yaml

# Full fine-tuning
python scripts/train_voxcpm_finetune.py \
    --config_path conf/voxcpm_v2/voxcpm_finetune_all.yaml

# WebUI for training & inference
python lora_ft_webui.py   # then open http://localhost:7860

Depending on your throughput and deployment needs, choose one of the following:

1. High Throughput (Nano-vLLM)

Use nano-vllm-voxcpm for optimized inference with support for concurrent requests and asynchronous APIs. It achieves significantly lower RTF (Real-Time Factor) on NVIDIA hardware.

2. Multi-tenant/OpenAI Compatible (vLLM-Omni)

Use vllm-omni for production-grade deployment. It supports PagedAttention, continuous batching, and provides an OpenAI-compatible /v1/audio/speech endpoint.

3. Edge/Consumer Hardware (llama.cpp-omni)

Use llama.cpp-omni for C++ based inference on CPU, Metal, CUDA, or Vulkan without a Python runtime. Requires GGUF weights (BaseLM + Acoustic).

# Nano-vLLM deployment
pip install nano-vllm-voxcpm

# vLLM-Omni deployment (OpenAI compatible)
# 1. Install vllm-omni from source
uv pip install vllm==0.19.0 --torch-backend=auto
git clone https://github.com/vllm-project/vllm-omni.git && cd vllm-omni
uv pip install -e .
# 2. Serve
vllm serve openbmb/VoxCPM2 --omni --port 8000

# llama.cpp-omni (Edge/C++)
git clone https://github.com/tc-mb/llama.cpp-omni.git && cd llama.cpp-omni
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --target voxcpm2-cli -j
./build/bin/voxcpm2-cli -t "Hello" -o out.wav VoxCPM2-BaseLM-Q8_0.gguf VoxCPM2-Acoustic-F16.gguf

VoxCPM supports both Full Parameter Fine-Tuning (SFT) and LoRA fine-tuning. LoRA is recommended for efficiency, requiring only 5-10 minutes of audio data to adapt to a specific speaker, language, or domain.

To perform LoRA fine-tuning, use the train_voxcpm_finetune.py script with a LoRA configuration file. For full parameter fine-tuning, use the same script with a full parameter configuration file.

You can also use a WebUI for training and inference by running lora_ft_webui.py and accessing the interface at http://localhost:7860.

# LoRA fine-tuning (parameter efficient, recommended)
python scripts/train_voxcpm_finetune.py \
    --config_path conf/voxcpm_v2/voxcpm_finetune_lora.yaml

# Full parameter fine-tuning
python scripts/train_voxcpm_finetune.py \
    --config_path conf/voxcpm_v2/voxcpm_finetune_all.yaml

# WebUI training and inference
python lora_ft_webui.py   # then open http://localhost:7860