OmniVoice Documentation

repository·master·Indexed 27 days ago

OmniVoice is a massively multilingual zero-shot text-to-speech (TTS) model supporting over 600 languages. It enables voice cloning via reference audio, voice design through speaker attributes (gender, age, pitch, style, and accent), and fine-grained control over non-verbal expressions and pronunciation. The library provides tools for single and batch inference, a Gradio web UI, and advanced data preparation scripts for audio denoising and tokenization.

Tokens: 11.4K
Snippets: 32
Records: 58
Agent score: 93%

What's inside OmniVoice

Reuse Cloned Voices with VoiceClonePrompt

master

To avoid re-processing reference audio in every session, encode the reference audio into a prompt and save it to disk.

from omnivoice import OmniVoice, VoiceClonePrompt

# 1. Create and save the prompt
prompt = model.create_voice_clone_prompt(
    ref_audio="ref.wav", 
    ref_text="Transcription of the reference audio."
)
prompt.save("my_voice.pt")

# 2. Later, load and use the prompt
prompt = VoiceClonePrompt.load("my_voice.pt")
audio = model.generate(text="Hello again!", voice_clone_prompt=prompt)

Use Non-Verbal Symbols and Pronunciation Control
master
OmniVoice allows fine-grained control over expression and pronunciation using inline tags in the input text.
Non-Verbal Symbols
Insert tags like [laughter] to add expressive sounds. Supported tags include:
- [laughter]
- [sigh]
- [confirmation-en]
- [question-en], [question-ah], [question-oh], [question-ei], [question-yi]
- [surprise-ah], [surprise-oh], [surprise-wa], [surprise-yo]
- [dissatisfaction-hnn]
Example:
```
audio = model.generate(text="[laughter] You really got me.")
```
Pronunciation Control
- Chinese: Use pinyin with tone numbers (e.g., ZHE2, SHE2) to correct character pronunciation.
- English: Use the CMU pronunciation dictionary format (uppercase, in brackets) to override default pronunciations.
Example (English):
```
audio = model.generate(text="He plays the [B EY1 S] guitar.")
```
https://github.com/k2-fsa/omnivoice/blob/master/README.md
Monitor training with TensorBoard
master
Training logs are written to TensorBoard. To view them, run the following command pointing to your output directory's tensorboard subdirectory:
```
tensorboard --logdir exp/omnivoice_emilia/tensorboard
```
https://github.com/k2-fsa/omnivoice/blob/master/docs/training.md
Install OmniVoice via pip
master
To install OmniVoice, first install PyTorch compatible with your hardware, then install the package. It is recommended to use a fresh virtual environment.
1. Install PyTorch
NVIDIA GPU (CUDA 12.8 example):
```
pip install torch==2.8.0+cu128 torchaudio==2.8.0+cu128 --extra-index-url https://download.pytorch.org/whl/cu128
```
Apple Silicon:
```
pip install torch==2.8.0 torchaudio==2.8.0
```
Intel Arc GPU (XPU):
1. Install Intel GPU drivers.
2. Install PyTorch with XPU support:
```
pip install torch torchaudio --index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
```
1. Verify with:
```
python -c "import torch; print(torch.xpu.is_available(), torch.xpu.device_count())"
```
2. Install OmniVoice
Choose one method:
Stable release from PyPI:
```
pip install omnivoice
```
Latest source from GitHub:
```
pip install git+https://github.com/k2-fsa/OmniVoice.git
```
Development (editable install):
```
git clone https://github.com/k2-fsa/OmniVoice.git
cd OmniVoice
pip install -e .
```
```
pip install omnivoice
```
https://github.com/k2-fsa/omnivoice/blob/master/README.md
Install OmniVoice with evaluation dependencies
master
To run evaluations, you must install the eval extra. Use pip or uv depending on your environment preference.
```
pip install omnivoice[eval]
# or
uv sync --extra eval
```
https://github.com/k2-fsa/omnivoice/blob/master/docs/evaluation.md
Tokenize audio with noise augmentation
master
To make the model robust to noisy reference audio during inference, you can add environmental noise and room reverb (RIR) to the prompt audio during the tokenization stage. This is performed using the omnivoice.scripts.extract_audio_tokens_add_noise module.
Prerequisites
You need two additional datasets in WebDataset format, each with its own data.lst manifest:
1. Noise recordings: Environmental noise tar shards.
2. Room impulse responses (RIR): RIR tar shards.
Usage
Run the omnivoice.scripts.extract_audio_tokens_add_noise module. You can use --input_jsonl for a JSONL manifest or --input_manifest for a custom WebDataset format dataset.
```
export CUDA_VISIBLE_DEVICES="0,1,2,4"
python -m omnivoice.scripts.extract_audio_tokens_add_noise \
    --input_jsonl data.jsonl \
    --tar_output_pattern data/tokens/shard-%06d.tar \
    --jsonl_output_pattern data/txts/shard-%06d.jsonl \
    --tokenizer_path eustlb/higgs-audio-v2-tokenizer \
    --noise_manifest data/noise_shards/data.lst \
    --rir_manifest data/rir_shards/data.lst \
    --nj_per_gpu 3
```
https://github.com/k2-fsa/omnivoice/blob/master/docs/data_preparation_advanced.md

Launch OmniVoice training

master

Use accelerate launch to start training. You must specify the training config, data config, and output directory via command line flags.

accelerate launch \
    --gpu_ids "0,1,2,3,4,5,6,7" \
    --num_processes 8 \
    -m omnivoice.cli.train \
    --train_config config/train_config_emilia.json \
    --data_config config/data_config_emilia.json \
    --output_dir exp/omnivoice_emilia

Convert audio to WebDataset shards using extract_audio_tokens.py

master

Use the extract_audio_tokens.py script to convert raw audio files into 8-layer discrete tokens packed into WebDataset tar shards. This process generates .tar files for audio tokens, companion .jsonl files for metadata, and a data.lst manifest.

From a raw JSONL file

export CUDA_VISIBLE_DEVICES="0,1,2,4"
python -m omnivoice.scripts.extract_audio_tokens \
    --input_jsonl data.jsonl \
    --tar_output_pattern output/audios/shard-%06d.tar \
    --jsonl_output_pattern output/txts/shard-%06d.jsonl \
    --tokenizer_path eustlb/higgs-audio-v2-tokenizer \
    --nj_per_gpu 3 \
    --shuffle True

From an existing WebDataset (raw-audio tar shards)

If you already have audio packed into tar shards, use jsonl_to_webdataset first to create a data.lst manifest, then pass that manifest to the extraction script:

Create manifest from JSONL:

python -m omnivoice.scripts.jsonl_to_webdataset \
    --input data.jsonl \
    --output data/shards \
    --sr 24000 \
    --shard-size 1000

Extract tokens using the manifest:

export CUDA_VISIBLE_DEVICES="0,1,2,4"
python -m omnivoice.scripts.extract_audio_tokens \
    --input_manifest existing_data/data.lst \
    --tar_output_pattern output/audios/shard-%06d.tar \
    --jsonl_output_pattern output/txts/shard-%06d.jsonl \
    --tokenizer_path eustlb/higgs-audio-v2-tokenizer \
    --nj_per_gpu 3 \
    --shuffle True

Prepare input JSONL for OmniVoice
master
Before processing, create a JSONL manifest where each line is a JSON object representing a single audio sample. This file serves as the source for tokenization.
Required fields:
- id: A unique sample identifier.
- audio_path: The absolute path to the audio file (supports wav, flac, or mp3; will be resampled to 24 kHz).
- text: The transcript text.
- language_id (optional): Language code for multilingual training.
Example format:
```
{"id": "sample_001", "audio_path": "/data/audio/001.wav", "text": "Hello world", "language_id": "en"}
{"id": "sample_002", "audio_path": "/data/audio/002.wav", "text": "你好世界", "language_id": "zh"}
```
https://github.com/k2-fsa/omnivoice/blob/master/docs/data_preparation.md
Synthesize Min Nan Chinese (Hokkien)
master
When synthesizing Min Nan Chinese (闽南语, also known as Hokkien), you must use Tai-lo romanization as the input. The current model version does not support Chinese characters for Min Nan Chinese synthesis.
https://github.com/k2-fsa/omnivoice/blob/master/docs/tips.md
Resume or Initialize training from checkpoints
master
You can control how training starts by adding specific keys to your training JSON config file.
Resume Training
To resume from an existing checkpoint, set resume_from_checkpoint to the path of the checkpoint directory:
```
{
    "resume_from_checkpoint": "exp/omnivoice/checkpoint-100000"
}
```
Initialize from Pretrained Model (Fine-tuning)
To start training from a pretrained OmniVoice checkpoint for fine-tuning, set init_from_checkpoint to the path of the checkpoint directory:
```
{
    "init_from_checkpoint": "exp/omnivoice/checkpoint-100000"
}
```
https://github.com/k2-fsa/omnivoice/blob/master/docs/training.md
Generate short audio clips reliably
master
The model may struggle to reliably generate short audio clips (e.g., 1–2 seconds) if no reference audio is provided. To ensure successful generation of short clips, always provide a ref_audio to the model.
https://github.com/k2-fsa/omnivoice/blob/master/docs/tips.md

OmniVoice Documentation

What's inside OmniVoice

Non-Verbal Symbols

Pronunciation Control

1. Install PyTorch

2. Install OmniVoice

Prerequisites

Usage

From a raw JSONL file

From an existing WebDataset (raw-audio tar shards)

Resume Training

Initialize from Pretrained Model (Fine-tuning)