DOCSEARCHbyAlgolia
Submit Docs

VieNeu-TTS Documentation

repository·main·Indexed 24 days ago

https://github.com/pnnbao97/vieneu-tts

A high-fidelity, bilingual (Vietnamese-English) text-to-speech system designed for on-device use. Featuring the v3 Turbo architecture, it supports 48 kHz audio, instant voice cloning with 3–5 seconds of reference audio, and multi-speaker conversation modes. The system includes a Python SDK (vieneu) with support for both ONNX (CPU) and PyTorch (GPU) engines, LoRA fine-tuning capabilities, and a Web UI.

Tokens
21.5K
Snippets
55
Records
135
Agent score
80%

What's inside VieNeu-TTS

  1. Overview of VieNeu-TTS

    main

    VieNeu-TTS is a high-fidelity, on-device Vietnamese Text-to-Speech (TTS) model. The latest version, VieNeu-TTS-v2, features a bilingual (English-Vietnamese) architecture trained on over 10,000 hours of data.

    Key capabilities include:

    • Bilingual Code-switching: Smooth transitions between English and Vietnamese within a single sentence.
    • Zero-shot Voice Cloning: Clone any voice using only 3-5 seconds of audio samples.
    • Podcast & Dialogue Modes: Supports multi-speaker interactions with automatic character recognition.
    • High Performance: Optimized for GPU (via LMDeploy) and CPU (via GGUF/ONNX).
    • Offline Capability: Generates high-quality 24 kHz audio completely offline.

  2. Overview of VieNeu-TTS Features

    main

    VieNeu-TTS is a high-fidelity Vietnamese Text-to-Speech system with the following core capabilities:

    • Bilingual (En-Vi) Code-switching: Seamlessly transitions between Vietnamese and English using sea-g2p.
    • Instant Voice Cloning: Clone voices using only 3–5 seconds of reference audio.
    • Podcast & Conversation Mode: Supports multi-speaker dialogue with automatic character detection.
    • v3 Turbo Architecture:
      • 48 kHz high-fidelity audio.
      • Built-in default voices (no reference clip needed).
      • Reading styles: natural, news, and storytelling.
      • Experimental Emotion Cues: Supports non-verbal cues like [cười], [thở dài], and [hắng giọng] directly in text.
      • Batched Generation: Supports batch sizes up to 32, including multi-speaker conversation batching.

  3. Configure reading styles and emotion cues

    main

    You can customize the delivery of the speech using style and inline emotion tags.

    Reading Styles

    Pass the style argument to infer() (defaults to "tu_nhien"):

    • "tu_nhien": Natural / conversational
    • "tin_tuc": News
    • "doc_truyen": Storytelling

    Emotion Cues

    Use inline tags within your text string to trigger specific expressions:

    • [cười]: Chuckle
    • [thở dài]: Sigh
    • [hắng giọng]: Clear throat"

  4. Model Backbone and Precision Specifications

    main

    VieNeu-TTS uses different backbones and precision settings depending on your hardware:

    ModelEngineDeviceSample RateFeatures
    VieNeu-TTS v3 Turbo (default)ONNX (CPU) / PyTorch (GPU)CPU/GPU48 kHzPreset voices, cloning, emotions

    Precision and Performance

    • On CPU: The backbone runs in int8 precision by default for maximum speed. If you require maximum quality, use Vieneu(precision="fp32").
    • On GPU (CUDA): Inference automatically uses batching. The API remains the same, but performance is optimized for GPU hardware.

  6. Clone voices using Zero-shot cloning

    main

    You can clone any voice using a short reference audio clip (3–8 seconds).

    Direct Cloning

    Use ref_audio in the infer() method. The SDK automatically denoises and clips the audio if denoise=True (default).

    Managing Cloned Voices

    To avoid re-processing the same clip, use add_voice() to register a voice. You can then call it by name in subsequent infer() calls. Use save_voices() to persist these profiles.

    Note: Voice cloning and add_voice currently require the PyTorch (GPU) engine.

  7. Deploy VieNeu-TTS as a high-performance API Server with Docker

    main

    You can deploy VieNeu-TTS as a high-performance API server (powered by LMDeploy) using Docker. This is recommended for production or remote usage.

    Requirements: You must have the NVIDIA Container Toolkit installed to support GPU acceleration.

    Standard Deployment

    Run the following command to start the server with a public tunnel (using bore), which allows you to access the server without opening modem ports:

    docker run --gpus all -p 23333:23333 -v huggingface_cache:/root/.cache/huggingface pnnbao/vieneu-tts:latest --tunnel
    • Default Model: The server defaults to VieNeu-TTS-v2 for maximum quality.
    • Tunneling: The Docker image includes bore. Check the container logs to find your public address (e.g., bore.pub:31631).

    Custom Model Deployment

    To run a specific model version (e.g., the faster 0.3B model):

    docker run --gpus all pnnbao/vieneu-tts:serve --model pnnbao-ump/VieNeu-TTS-0.3B --tunnel

    To serve a locally fine-tuned model (e.g., after merging a LoRA adapter), mount your output directory into the container:

    # Linux / macOS
docker run --gpus all \
  -v $(pwd)/finetune/output:/workspace/models \
  pnnbao/vieneu-tts:serve \
  --model /workspace/models/merged_model --tunnel

  8. Use LoRA Adapters with LMDeploy for high-speed fine-tuned voices

    main

    To run fine-tuned voices with optimal performance, use the LoRA Adapter method with LMDeploy acceleration. This process involves configuring a custom model ID, downloading/merging the adapter with a base model, and providing reference audio/text.

    Configuration Steps

    1. In the 🦜 Backbone section, select Custom Model.
    2. Custom Model ID: Enter the HuggingFace Repo ID or a local folder path (e.g., pnnbao-ump/VieNeu-TTS-0.3B-lora-ngoc-huyen).
    3. HF Token (Optional): Enter your HuggingFace Access Token if the repository is Private.
    4. Base Model: Select the corresponding Base Model that the LoRA was trained on (e.g., if the LoRA name contains "0.3", select the 0.3B base model).
    5. 🚀 Optimize with LMDeploy: Check this box to enable high-speed acceleration.

    Loading the Model

    1. Click 🔄 Tải Model (Download Model).
    2. The system will automatically download the LoRA Adapter and Base Model, merge them (using GPU if available), and save the result to the merged_models_cache/ directory.
    3. The merged model is then loaded using LMDeploy.

    Generating Custom Voice

    1. Navigate to the "🦜 Custom Voice" tab.
    2. Audio Reference: Upload a sample audio file (ideally from the LoRA training set).
    3. Text Reference: Enter the exact text content of the uploaded audio file, matching punctuation and wording perfectly.
    4. Enter your target text and click 🎵 Bắt đầu (Start).

  9. Install `make` on macOS or Linux

    main

    macOS

    • Homebrew: brew install make
    • Xcode Tools: Run xcode-select --install in your terminal.

    Linux

    • Ubuntu/Debian: sudo apt update && sudo apt install make
    • Arch Linux: sudo pacman -S make
    • Fedora: sudo dnf install make
    sudo apt update && sudo apt install make

  10. Deploy VieNeu-TTS as a High-Quality API Server with Docker

    main

    You can deploy VieNeu-TTS as a high-performance API server using Docker. This is recommended for production environments. The server is powered by LMDeploy and can be exposed via a public tunnel using the built-in bore tool, which eliminates the need for manual port forwarding.

    Requirements:

    Default Behavior:

    • Loads the VieNeu-TTS-v2 model for maximum quality.
    • Uses the --tunnel flag to provide a public address (e.g., bore.pub:31631). Check container logs to find your specific address.
    docker run --gpus all -p 23333:23333 -v huggingface_cache:/root/.cache/huggingface pnnbao/vieneu-tts:latest --tunnel

  11. Install VieNeu-TTS using `uv` (Recommended)

    main

    The recommended way to manage dependencies for VieNeu-TTS is using uv.

    1. Install uv

    Windows:

    powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

    Linux/macOS:

    curl -LsSf https://astral.sh/uv/install.sh | sh

    2. Clone the Repository

    git clone https://github.com/pnnbao97/VieNeu-TTS.git
cd VieNeu-TTS

    3. Install Dependencies

    Choose one of the following two options based on your hardware:

    Option 1: CPU & macOS (Minimal, no PyTorch) Recommended for maximum speed on CPU using v3 Turbo with ONNX. This avoids installing PyTorch and uses a lightweight ONNX Runtime. For macOS users, the ONNX path on CPU is often faster than MPS/PyTorch for v3 Turbo.

    uv sync

    Option 2: GPU (NVIDIA CUDA or Apple Silicon) Use this to run v3 Turbo on GPU (PyTorch). Requires NVIDIA CUDA (≥ 12.8) or Apple Silicon MPS. On CUDA, it supports automatic batch inference.

    uv sync --group gpu
    uv sync
# OR
uv sync --group gpu

  12. Deploy as a Remote Server (One-Command)

    main

    To provide a 'one-command' experience for users where they can run a pre-built server image without cloning the repository, use the serve image.

    For Maintainers: Build and Push

    Use the provided Makefile targets to build and push the optimized server image (defaults to pnnbao97/vieneu-tts:serve):

    docker login
make docker-build-serve
make docker-push-serve

    For Users: Run the Server

    Any user with an NVIDIA GPU can run the optimized API server with this command:

    docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  pnnbao97/vieneu-tts:serve