VibeVoice

VibeVoice-Realtime is a lightweight (0.5B parameter) text-to-speech model designed for low-latency, long-form speech generation.

Key Characteristics:

• Streaming Input: Supports incremental text chunks, allowing LLMs to start speaking from their first tokens.
• Interleaved Windowed Design: It encodes incoming text chunks in parallel with diffusion-based acoustic latent generation from prior context.
• Acoustic Tokenizer: Unlike full multi-speaker variants, this model removes the semantic tokenizer and uses an efficient acoustic tokenizer operating at 7.5 Hz.
• Context Window: Supports an 8k context window, capable of generating approximately 10 minutes of audio.
• Single Speaker: This specific real-time variant supports only a single speaker. For multi-speaker conversational needs, use the long-form multi-speaker variants of VibeVoice.

Limitations:

• Language: Primarily English. Other languages may produce unpredictable results.
• Content: Does not support code, mathematical formulas, or special symbols. Pre-process text to normalize these.
• Audio Type: Focused on speech only; does not handle background noise or music.
• Input Length: Extremely short inputs (3 words or fewer) may cause stability degradation.

To deploy the VibeVoice ASR server, use a Docker container based on vllm/vllm-openai:v0.14.1. The launcher script start_server.py handles system dependencies, model downloads, and server initialization.

Single GPU Setup

Use this command to run the server on a single GPU (default behavior):

docker run -d --gpus '"device=0"' --name vibevoice-asr-demo \
  --ipc=host \
  -p 6001:6001 \
  -e VIBEVOICE_FFMPEG_MAX_CONCURRENCY=64 \
  -e PYTORCH_ALLOC_CONF=expandable_segments:True \
  -v $(pwd):/app \
  -w /app \
  --entrypoint bash \
  vllm/vllm-openai:v0.14.1 \
  -c "python3 /app/vllm_plugin/scripts/start_server.py --port 6001"

Multi-GPU with Data Parallel (Load Balancing)

To scale throughput, run multiple replicas using the --dp N flag. For example, to run 4 replicas across 4 GPUs:

docker run -d --gpus '"device=0,1,2,3"' --name vibevoice-asr-demo \
  --ipc=host \
  -p 6001:6001 \
  -e VIBEVOICE_FFMPEG_MAX_CONCURRENCY=64 \
  -e PYTORCH_ALLOC_CONF=expandable_segments:True \
  -v $(pwd):/app \
  -w /app \
  --entrypoint bash \
  vllm/vllm-openai:v0.14.1 \
  -c "python3 /app/vllm_plugin/scripts/start_server.py --port 6001 --dp 4"

Note: Use --tp N for tensor parallel if working with very large models.

The VibeVoice launcher supports two types of GPU parallelism to optimize for either throughput or model size:

• Data Parallel (--dp N): Runs N independent model replicas across N GPUs. This is recommended for scaling throughput. An nginx reverse proxy automatically load balances requests across the workers.
• Tensor Parallel (--tp N): Splits a single model across N GPUs. Use this if the model is too large to fit in the memory of a single GPU.
• Hybrid (--dp N --tp M): Combines both strategies. The total GPUs required is N * M.

Note: Ensure you expose the correct number of devices in the Docker --gpus flag.

# Data Parallel: 4 independent replicas on 4 GPUs
docker run -d --gpus '"device=0,1,2,3"' --name vibevoice-vllm \
  --ipc=host -p 8000:8000 -v $(pwd):/app -w /app \
  --entrypoint bash vllm/vllm-openai:v0.14.1 \
  -c "python3 /app/vllm_plugin/scripts/start_server.py --dp 4"

# Tensor Parallel: 1 model split across 2 GPUs
docker run -d --gpus '"device=0,1"' --name vibevoice-vllm \
  --ipc=host -p 8000:8000 -v $(pwd):/app -w /app \
  --entrypoint bash vllm/vllm-openai:v0.14.1 \
  -c "python3 /app/vllm_plugin/scripts/start_server.py --tp 2"

# Hybrid: 2 replicas, each split across 2 GPUs (4 GPUs total)
docker run -d --gpus '"device=0,1,2,3"' --name vibevoice-vllm \
  --ipc=host -p 8000:8000 -v $(pwd):/app -w /app \
  --entrypoint bash vllm/vllm-openai:v0.14.1 \
  -c "python3 /app/vllm_plugin/scripts/start_server.py --dp 2 --tp 2"

To use VibeVoice-Realtime, it is recommended to use an NVIDIA Deep Learning Container to manage the CUDA environment.

1. Launch Docker: Use an NVIDIA PyTorch Container (versions 24.07, 24.10, or 24.12 are verified; later versions are compatible).
2. Install Flash Attention: If your Docker environment does not include flash-attn, install it manually using pip install flash-attn --no-build-isolation.
3. Install VibeVoice: Clone the repository and install the package with the [streamingtts] extra.

Note: The model is primarily designed for English. While it shows some multilingual capability in German, French, Italian, Japanese, Korean, Dutch, Polish, Portuguese, and Spanish, these have not been extensively tested.

# 1. Launch docker
sudo docker run --privileged --net=host --ipc=host --ulimit memlock=-1:-1 --ulimit stack=-1:-1 --gpus all --rm -it  nvcr.io/nvidia/pytorch:24.07-py3

# 2. Install flash-attn if not present
pip install flash-attn --no-build-isolation

# 3. Install from github
git clone https://github.com/microsoft/VibeVoice.git
cd VibeVoice/
pip install -e .[streamingtts]