Retrieval-based Voice Conversion

repository·develop·Indexed 17 days ago

https://github.com/rvc-project/retrieval-based-voice-conversion

A voice conversion framework based on VITS (version 0.3.5) that transforms input audio into a target voice using pre-trained models and feature indexing. It provides a Python library via the VC class, a REST API server, and a comprehensive CLI for inference, training, environment configuration, and UVR5 vocal/instrumental separation.

Tokens
4.2K
Snippets
16
Records
17
Agent score
70%

What's inside RVC

  1. Run RVC using Docker

    develop

    Quick Run

    You can build and run the container using the provided script:

    ./docker-run.sh

    Manual Docker Setup

    If running manually, ensure your weights, indices, and input audios are stored in the ${PWD}/assets directory.

    1. Build the image:
    docker build -t "rvc" .
    1. Run the container: Map the necessary asset directories as read-only volumes and expose port 8000.
    docker run -it \
      -p 8000:8000 \
      -v "${PWD}/assets/weights:/weights:ro" \
      -v "${PWD}/assets/indices:/indices:ro" \
      -v "${PWD}/assets/audios:/audios:ro" \
      "rvc"
    docker build -t "rvc" .
    docker run -it \
      -p 8000:8000 \
      -v "${PWD}/assets/weights:/weights:ro" \
      -v "${PWD}/assets/indices:/indices:ro" \
      -v "${PWD}/assets/audios:/audios:ro" \
      "rvc"
  2. Configure RVC environment and download models

    develop

    Custom Configuration

    If you need to change model locations or configurations, edit the .env file. If you don't have one, generate it using:

    rvc env create

    Downloading Models

    You can download models directly via the CLI:

    • Download to default location: rvc dlmodel
    • Download to a specific directory: rvc dlmodel {download_dir}

    After downloading, ensure the model location is correctly specified in your .env file.

    rvc env create
    rvc dlmodel {download_dir}
  3. Perform audio inference using the RVC library

    develop

    You can use the VC class from rvc.modules.vc.modules to perform voice conversion programmatically.

    1. Initialize the VC object.
    2. Load the model using get_vc(model_path).
    3. Run inference with vc_inference(sid, input_audio_path) which returns the target sample rate, audio data, and timing information.
    from pathlib import Path
    from dotenv import load_dotenv
    from scipy.io import wavfile
    from rvc.modules.vc.modules import VC
    
    
    def main():
          vc = VC()
          vc.get_vc("{model.pth}")
          tgt_sr, audio_opt, times, _ = vc.vc_inference(
                1, Path("{InputAudio}")
          )
          wavfile.write("{OutputAudio}", tgt_sr, audio_opt)
    
    
    if __name__ == "__main__":
          load_dotenv("{envPath}")
          main()
  4. Use the RVC API for audio inference

    develop

    Start the API Server

    Run the server using:

    rvc-api

    Inference Endpoints

    All inference requests are POST requests to /inference using multipart/form-data. You can specify the response type via the res_type query parameter.

    Get response as blob

    Use res_type=blob to receive the raw audio data.

    Get response as JSON

    Use res_type=json to receive a JSON object containing the audio and timing information.

    # Get as blob
    curl -X 'POST' \
          'http://127.0.0.1:8000/inference?res_type=blob' \
          -H 'accept: application/json' \
          -H 'Content-Type: multipart/form-data' \
          -F 'modelpath={model.pth}' \
          -F 'input={input audio path}'
    
    # Get as json(include time)
    curl -X 'POST' \
          'http://127.0.0.1:8000/inference?res_type=json' \
          -H 'accept: application/json' \
          -H 'Content-Type: multipart/form-data' \
          -F 'modelpath={model.pth}' \
          -F 'input={input audio path}'
  5. Reference: rvc infer CLI options

    develop

    The following options are available for the rvc infer command:

    | option        | flag | type         | default value | description |
    |---------------|------------|--------------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
    | modelPath     | -m         | Path         | *required     | Model path or filename (reads in the directory set in env) |
    | inputPath     | -i         | Path         | *required     | Input audio path or folder |
    | outputPath    | -o         | Path         | *required     | Output audio path or folder |
    | sid           | -s         | int          | 0             | Speaker/Singer ID |
    | f0_up_key     | -fu        | int          | 0             | Transpose (integer, number of semitones, raise by an octave: 12, lower by an octave: -12) |
    | f0_method     | -fm        | str          | rmvpe         | pitch extraction algorithm (pm, harvest, crepe, rmvpe) |
    | f0_file       | -ff        | Path \| None | None          | F0 curve file (optional). One pitch per line. Replaces the default F0 and pitch modulation |
    | index_file    | -if        | Path \| None | None          | Path to the feature index file |
    | index_rate    | -if        | float        | 0.75          | Search feature ratio (controls accent strength, too high has artifacting) |
    | filter_radius | -fr        | int          | 3             | If >=3: apply median filtering to the harvested pitch results. The value represents the filter radius and can reduce breathiness |
    | resample_sr   | -rsr       | int          | 0             | Resample the output audio in post-processing to the final sample rate. Set to 0 for no resampling |
    | rms_mix_rate  | -rmr       | float        | 0.25          | Adjust the volume envelope scaling. Closer to 0, the more it mimicks the volume of the original vocals. Closer to 1 will be a more of a consistently loud volume |
    | protect       | -p         | float        | 0.33          | Protect voiceless consonants and breath sounds to prevent artifacts such as tearing in electronic music. Set to 0.5 to disable. Decrease the value to increase protection, but it may reduce indexing accuracy |
  6. RVC CLI command reference

    develop

    The following commands are registered under the rvc CLI group:

    • infer: Handles voice conversion/inference tasks.
    • train: Manages the training process for voice models.
    • uvr: Provides UVR5 (Ultimate Vocal Remover) functionality for vocal/instrumental separation.
    • dlmodel: Utilities for handling deep learning models.
    • env: Manages environment-specific configurations or settings.
    • init: Performs initialization tasks for the RVC environment.
  7. Reference: `infer` CLI options

    develop

    Full list of command-line options available for the infer command.

    --modelPath, -m (str, required): Model path or filename (reads in the directory set in env)
    --inputPath, -i (Path, required): input audio path or folder
    --outputPath, -o (Path, required): output audio path or folder
    --sid, -s (int): Speaker/Singer id (default: 0)
    --f0upkey, -fu (int): Transpose (default: 0)
    --f0method, -fm (str): Pitch extraction algorith (default: "rmvpe")
    --f0file, -ff (Path): F0 curve file (optional)
    --indexFile, -if (Path): Feature index file
    --indexRate, -ir (float): Search feature ratio (default: 0.75)
    --filterRadius, -fr (int): Apply median filtering (default: 3)
    --resamplesr, -rsr (int): Resample the output audio (default: 0)
    --rmsmixrate, -rmr (float): Adjust the volume envelope scaling (default: 0.25)
    --protect, -p (float): Protect voiceless consonants and breath sounds (default: 0.33)
  8. Reference UVR5 CLI options

    develop

    The following options are available for the uvr command:

    OptionLong FlagTypeDescription
    -m--modelNamestrModel path or filename (reads in the directory set in env)
    -i--inputPathPathInput audio path or folder
    -o--outputPathPathRequired. Output audio path or folder
    -f--formatstrOutput Format
    -m, --modelName STR
        Model path or filename (reads in the directory set in env)
    -i, --inputPath PATH
        input audio path or folder
    -o, --outputPath PATH
        output audio path or folder (REQUIRED)
    -f, --format STR
        output Format
  9. Use the UVR5 CLI command for audio separation

    develop

    The uvr command is used to perform audio separation using UVR5 models. It accepts an input audio file or folder, a model name/path, and an output destination. The command uses the UVR module to process the files and can specify an output format.

    Note: The --modelName option reads from the directory specified in your environment variables if a full path is not provided.

    # Example usage (placeholders for actual paths)
    python -m rvc.wrapper.cli.handler.uvr5 --modelName "model_name" --inputPath "/path/to/input" --outputPath "/path/to/output" --format "wav"