DOCSEARCHbyAlgolia
Submit Docs

RVC (Retrieval-based Voice Conversion)

repository·main·Indexed 13 days ago

https://github.com/rvc-project/retrieval-based-voice-conversion-webui

A high-quality voice conversion and voice changer framework featuring a web-based interface for training and real-time inference. It includes support for RMVPE and FCPE pitch extraction, PyMSS vocal separation, and a decoupled RVC Realtime VST plugin for low-latency audio processing on Windows x64.

Tokens
25.6K
Snippets
55
Records
153
Agent score
95%

What's inside RVC

  1. Understand Faiss usage in RVC

    main
    RVC uses the faiss library (developed by Facebook Research) to perform approximate nearest neighbor search on HuBERT feature embeddings. Instead of performing an exhaustive search through all training data embeddings—which is computationally expensive—RVC uses approximate methods to quickly find similar embeddings and mix them during voice conversion, resulting in faster inference.

  2. RVC Model and Directory Structure

    main

    The WebUI expects models and assets to be placed in specific directories. User-provided models should be placed in assets/weights/ and .index files in assets/indices/.

    Required Directory Structure:

    • assets/hubert_base/ (contains config.json, preprocessor_config.json, pytorch_model.bin)
    • assets/rmvpe/ (contains rmvpe.pt or rmvpe.onnx)
    • assets/pretrained/ (v1 pretrained models)
    • assets/pretrained_v2/ (v2 pretrained models)
    • assets/pymss_weights/ (vocal separation weights)
    • assets/weights/ (User .pth models)
    • assets/indices/ (User .index files)
    • logs/mute/ (training silence samples)

  3. Optimize Pitch Extraction with RMVPE

    main

    The RMVPE (Robust Model for Pitch Extraction) is the recommended pitch extraction algorithm. It is highly effective, especially for low-pitched male voices, and supports pytorch, onnx, and DirectML.

    Hardware Support:

    • Supports GPU acceleration via onnx_dml for RMVPE inference.
    • For AMD/Intel cards, pytorch-dml provides support for inference and separation, though training currently falls back to CPU.

  4. Configure Hardware Acceleration and Precision

    main

    The system features automatic GPU recognition to determine the appropriate inference mode and precision.

    • AMD/Intel GPUs: Support for AMD and Intel cards is provided via pytorch-dml for real-time conversion, inference, and vocal separation. Note that training is not yet supported on these cards and will fallback to CPU. However, onnx_dml can be used for rmvpe_gpu inference on these platforms.

  7. Configure IVF (Inverted File) parameters for RVC

    main

    IVF (Inverted File Index) partitions the search space into clusters using k-means. During search, only a subset of clusters (n_probe) is searched.

    Tuning IVF for RVC:

    • Number of clusters (n_ivf): For datasets with $N$ data points (where $N ext{ is } ext{1M or less}$), it is recommended to set the number of clusters between $4\sqrt{N}$ and $16\sqrt{N}$. Setting it too high makes the search as inefficient as an exhaustive search.
    • Search probes (n_probe): This determines how many clusters are checked during a query. Increasing n_probe improves accuracy but increases calculation time. For RVC, n_probe = 1 is often sufficient as extreme precision is not required.

  9. Understand Faiss's role in RVC feature retrieval

    main
    In RVC, Faiss is used to perform approximate neighborhood searches on HuBERT feature embeddings. During conversion, the system searches for embeddings in the training data that are similar to the current input embedding to achieve a more natural voice conversion. Because a naive exhaustive search is slow, RVC uses Faiss to implement high-speed approximate searches.

  10. Choose between L2 Distance and Inner Product metrics

    main

    Faiss supports different metrics for measuring embedding similarity:

    1. Euclidean Distance (METRIC_L2): Calculates the square root of the sum of squared differences across all dimensions. This is standard for most spatial distance calculations.
    2. Inner Product (METRIC_INNER_PRODUCT): Often used as Cosine Similarity when vectors are L2-normalized. This is common in models like word2vec or ArcFace.

    To perform L2 normalization on a vector X using numpy (to prepare for Inner Product/Cosine Similarity), use the following pattern to avoid division by zero:

    # X_normed = X / max(eps, ||X||_2)
X_normed = X / np.maximum(eps, np.linalg.norm(X, ord=2, axis=-1, keepdims=True))
    X_normed = X / np.maximum(eps, np.linalg.norm(X, ord=2, axis=-1, keepdims=True))

  11. Understand Feature Indexing for Inference

    main

    During inference, RVC searches for feature values similar to those used during training to improve accuracy. To make this search fast, an index is pre-calculated.

    • Mechanism: Uses the faiss (approximate neighborhood search) library.
    • Source Data: Reads from /logs/your-experiment-name/3_feature256.
    • Output: Saves an index file as logs/your-experiment-name/add_XXX.index.
    • Note: In versions updated after 2023-04-28, the system handles reading the index automatically, so manual specification may no longer be necessary.

  12. Configure RVC models and runtime directories

    main

    RVC expects a specific directory structure for models and assets. The WebUI creates runtime directories automatically, but you must manually place downloaded models in the correct locations.

    Directory Layout:

    • assets/hubert_base/: Contains config.json, preprocessor_config.json, and pytorch_model.bin.
    • assets/rmvpe/: Contains rmvpe.pt (and rmvpe.onnx for Windows AMD/Intel DirectML).
    • assets/pretrained/: Pretrained v1 models.
    • assets/pretrained_v2/: Pretrained v2 models.
    • assets/pymss_weights/: Weights for pymss/MSST vocal separation.
    • assets/weights/: User RVC .pth models.
    • assets/indices/: User .index files.
    • logs/mute/: Training silence samples.

    Note for Windows AMD/Intel (DirectML): You must also download rmvpe.onnx into assets/rmvpe/.