DOCSEARCHbyAlgolia
Submit Docs

kNN-VC Documentation

repository·master·Indexed 19 days ago

https://github.com/bshall/knn-vc

An any-to-any voice conversion model that utilizes k-nearest neighbors regression to map source features to target features within a self-supervised WavLM feature space, followed by HiFiGAN vocoding. The library supports loading via torch.hub and provides tools for feature extraction, kNN matching, and training the HiFiGAN vocoder.

Tokens
1.8K
Snippets
7
Records
8
Agent score
18%

What's inside kNN-VC

  2. Run kNN-VC inference workflow

    master

    The inference process follows four steps: loading the model, computing features for the source and reference audio, performing kNN matching, and vocoding the result.

    Requirements for Reference Audio:

    • The target speaker in ref_wav_paths can be any clean speech from the desired speaker.
    • Longer cumulative duration of reference waveforms improves quality, though benefits diminish beyond 5 minutes.
    • All input waveforms should be 16kHz.

    Output:

    • out_wav is a (T,) tensor representing the converted 16kHz output waveform.
    import torch, torchaudio

# 1. Load models
knn_vc = torch.hub.load('bshall/knn-vc', 'knn_vc', prematched=True, trust_repo=True, pretrained=True)

# 2. Compute features
src_wav_path = '<path to arbitrary 16kHz waveform>.wav'
ref_wav_paths = ['<path to arbitrary 16kHz waveform from target speaker>.wav', '<path to 2nd utterance from target speaker>.wav', ...]

query_seq = knn_vc.get_features(src_wav_path)
matching_set = knn_vc.get_matching_set(ref_wav_paths)

# 3. Perform kNN matching and vocoding
out_wav = knn_vc.match(query_seq, matching_set, topk=4)

  3. Perform voice conversion using torch.hub

    master

    You can load the kNN-VC model directly via torch.hub without cloning the repository. The model consists of a WavLM encoder and a HiFiGAN vocoder.

    Use the prematched argument to choose between the high-performance prematched vocoder or a regular HiFiGAN vocoder. Setting prematched=True uses the best model described in the paper.

    import torch, torchaudio

# Load the model with the prematched vocoder (recommended)
knn_vc = torch.hub.load('bshall/knn-vc', 'knn_vc', prematched=True, trust_repo=True, pretrained=True)

# Alternatively, use the regular HiFiGAN vocoder
# knn_vc = torch.hub.load('bshall/knn-vc', 'knn_vc', prematched=False, trust_repo=True, pretrained=True)

  4. Train the HiFiGAN vocoder

    master

    To train the HiFiGAN vocoder for WavLM features, use the adapted training script hifigan/train.py. This requires dependencies like librosa, tensorboard, matplotlib, fastprogress, and scipy.

    python -m hifigan.train \
    --audio_root_path /path/to/librispeech/root/ \
    --feature_root_path /path/to/the/output/of/previous/step/ \
    --input_training_file data_splits/wavlm-hifigan-train.csv \
    --input_validation_file data_splits/wavlm-hifigan-valid.csv \
    --checkpoint_path /path/where/you/want/to/save/checkpoint \
    --fp16 False \
    --config hifigan/config_v1_wavlm.json \
    --stdout_interval 25 \
    --training_epochs 1800 \
    --fine_tuning

  5. Load the kNN-VC model via Torch Hub

    master

    You can load the kNN-VC model directly using torch.hub.load. To use the pre-trained model, set pretrained=True. If you want to use a vocoder that was not trained using prematched data, set prematched=False. Ensure you specify the appropriate device (e.g., 'cuda').

    import torch

knn_vc = torch.hub.load('bshall/knn-vc', 'knn_vc', prematched=True, trust_repo=True, pretrained=True, device='cuda')

  6. Perform voice conversion with kNN-VC

    master

    Voice conversion follows a three-step process:

    1. Extract features from the source waveform using get_features.
    2. Create a matching set from target speaker reference waveforms using get_matching_set.
    3. Perform the conversion using match.

    Requirements:

    • All input waveforms must be 16kHz and single-channel.
    • src_wav_path should be a string path to a single waveform.
    • ref_wav_paths should be a list of string paths to reference waveforms from the target speaker.
    # 1. Extract features from source
query_seq = knn_vc.get_features(src_wav_path)

# 2. Create matching set from target references
matching_set = knn_vc.get_matching_set(ref_wav_paths)

# 3. Match and convert
out_wav = knn_vc.match(query_seq, matching_set, topk=4)

  7. Precompute WavLM features with prematch_dataset.py

    master

    Before training the HiFiGAN vocoder, you must precompute WavLM features for your dataset. Use the prematch_dataset.py script. You can use the --prematch flag to determine whether to use prematching when generating features.

    # Example: Generate the dataset used to train the prematched HiFiGAN from the paper
python prematch_dataset.py \
    --librispeech_path /path/to/librispeech/root \
    --out_path /path/where/you/want/outputs/to/go \
    --topk 4 \
    --matching_layer 6 \
    --synthesis_layer 6 \
    --prematch

  8. Save or play the converted audio

    master

    The output of knn_vc.match is a tensor representing the converted waveform. You can play it in a notebook using IPython.display.Audio or save it to a file using torchaudio.save.

    # Play in notebook
import IPython.display as ipd
ipd.Audio(out_wav.numpy(), rate=16000)

# Save to file
import torchaudio
torchaudio.save('knnvc1_out.wav', out_wav[None], 16000)