MeanVC: Streaming Zero-Shot Voice Conversion

Create a training list file (e.g., train.list) where each line represents a training sample. The format must follow this pattern:

utterance_id|bn_path|mel_path|xvector_path|prompt_mel_path1|prompt_mel_path2|...

# Example format:
utt001|/path/to/bns/utt001.npy|/path/to/mels/utt001.npy|/path/to/xvectors/utt001.npy|/path/to/mels/prompt01.npy

To start training, modify the configurations in script/train.sh (specifically data paths and the model directory) and execute the script.

bash script/train.sh

MeanVC requires several pre-trained models to function.

1. Automatic Download: Run python download_ckpt.py to download the main VC model, vocoder, and ASR model into src/ckpt/.
2. Manual Download: The speaker verification model (wavlm_large_finetune.pth) must be downloaded manually from Google Drive and placed in src/runtime/speaker_verification/ckpt/.

To use MeanVC, clone the repository, create a Conda environment with Python 3.11, and install the required dependencies via pip.

# Clone the repository and enter the directory
git clone https://github.com/ASLP-lab/MeanVC.git
cd MeanVC

# Create and activate a Conda environment
conda create -n meanvc python=3.11 -y
conda activate meanvc

# Install dependencies
pip install -r requirements.txt

For batch processing or converting pre-recorded files, use the scripts/infer_ref.sh script.

Configuration: Before running, edit scripts/infer_ref.sh to set the following variables:

• source_path: Path to the source audio file or directory.
• reference_path: Path to a clean audio file of the target speaker.
• output_dir: Directory for saved files (default: src/outputs).
• steps: Number of denoising steps (default: 2).

bash scripts/infer_ref.sh

Before training, you must extract Mel spectrograms, content features (BN), and speaker embeddings from your raw audio data using the following sequence of commands:

# 1. Extract Mel spectrograms (10ms frame shift)
python src/preprocess/extrace_mel_10ms.py --input_dir path/to/wavs --output_dir path/to/mels

# 2. Extract content features (160ms window)
python src/preprocess/extract_bn_160ms.py --input_dir path/to/wavs --output_dir path/to/bns

# 3. Extract speaker embeddings
python src/preprocess/extract_spk_emb_wavlm.py --input_dir path/to/wavs --output_dir path/to/xvectors

To achieve lightweight streaming voice conversion, the DiT model utilizes a two-part caching strategy:

1. Mel Cache (cache): This stores previous mel spectrogram frames. The model uses self.cache_embed to project these frames into the model dimension and concatenates them with the current input x to provide temporal context.
2. KV Cache (kv_cache): This stores the Key and Value tensors from the attention layers within each ChunkDiTBlock. By passing the new_kv_cache returned from one forward call into the next, the model avoids recomputing attention for all previous tokens, enabling efficient real-time processing.

When is_inference=True, the model adjusts its Rotary Positional Embedding (rope) calculation using an offset to ensure positional encoding remains consistent across chunks.

Use the run_rt.py script to capture audio from your microphone and convert it to a target speaker's voice in real-time. You will be prompted to select your audio input and output devices during execution.

python src/runtime/run_rt.py --target-path "path/to/target_voice.wav"

You can switch between vocos and bigvgan extraction methods by setting the mel_spec_type argument in the MelSpec constructor.

# For Vocos backend
extractor_vocos = MelSpec(mel_spec_type="vocos")

# For BigVGAN backend
extractor_bigvgan = MelSpec(mel_spec_type="bigvgan")

The MelSpec module provides a unified interface for converting raw waveforms into Mel spectrograms using either the vocos or bigvgan backends. This is useful for preprocessing audio data for model input.

Supported backends:

• vocos: Uses torchaudio.transforms.MelSpectrogram.
• bigvgan: Uses a custom implementation based on BigVGAN's approach.

Parameters:

• n_fft: FFT size.
• hop_length: Hop length.
• win_length: Window length.
• n_mel_channels: Number of Mel channels.
• target_sample_rate: Target sampling rate.
• mel_spec_type: Either 'vocos' or 'bigvgan'.

from src.model.modules import MelSpec
import torch

# Initialize extractor
mel_extractor = MelSpec(mel_spec_type="vocos", n_mel_channels=100)

# Dummy waveform: [batch, channels, length]
waveform = torch.randn(1, 1, 24000)

# Extract mel spectrogram
mel_spec = mel_extractor(waveform)

The PromptEncoder is a simplified version of the VP encoder. It uses PromptEncoderLayer which performs self-attention on the prompt followed by a position-wise feed-forward network. It does not perform explicit cross-attention with VP features in its layers, but it still uses the vp feature to construct keys and values for the self-attention mechanism.

Input feature dictionary must contain:

• prompt: Tensor of shape (batch, time, dim)
• vp: Tensor of shape (batch, vp_dim)

encoder = PromptEncoder(
    n_head=4, 
    n_feat=1024, 
    dropout_rate=0.1, 
    q_in_dim=1024, 
    num_blocks=2
)

feature = {
    'prompt': torch.rand(2, 1024, 1024),
    'vp': torch.rand(2, 192)
}

output = encoder(feature)