DiffSinger

DiffSinger provides several Mel pipelines depending on the task (SVS vs TTS), dataset, and input type (MIDI vs Lyric).

Use this pipeline to avoid issues with explicit F0 prediction in the melody frontend (such as bad uv/v predictions). This method uses joint prediction with spectrograms.

Pipeline Flow:

1. Melody Frontend: [lyrics] + [MIDI] -> [linguistic representation] + [predicted phoneme duration]
2. Acoustic Model: [linguistic representation (with MIDI information)] + [predicted phoneme duration] -> [mel-spectrogram]
3. Pitch Extractor: [mel-spectrogram] -> [predicted F0]
4. Vocoder: [mel-spectrogram] + [predicted F0] -> [waveform]

For detailed instructions, refer to README-SVS-opencpop-e2e.md.

DiffSinger-PNDM requires a pre-trained HifiGAN-Singing vocoder and a specific pendant.

Before training your acoustic model, unzip the pre-trained vocoder and the pendant for vocoder into the checkpoints directory.

Expected directory structure:

checkpoints/
└── 0109_hifigan_bigpopcs_hop128 (
    ├── model_ckpt_steps_1512000.ckpt
    └── config.yaml
)

If you prefer using a standard Python virtual environment, ensure Python 3.8 is installed first. You must manually install pip, Cython, numpy, and torch before installing the project requirements.

## Install Python 3.8 first. 
python -m venv venv
source venv/bin/activate
# install requirements.
pip install -U pip
pip install Cython numpy==1.19.1
pip install torch==1.9.0
pip install -r requirements.txt

To use the Opencpop dataset with the DiffSinger-PNDM pipeline, follow these steps:

1. Acquire Data: Download and extract the Opencpop dataset following the official instructions.
2. Link Dataset: Create a symbolic link from your Opencpop folder to the project's raw data directory: ln -s /path/to/opencpop data/raw/
3. Binarize Data: Run the binarization script to generate the packed dataset (data/binary/opencpop-midi-dp) required for training and inference.

Ensure PYTHONPATH is set to the project root.

export PYTHONPATH=.
CUDA_VISIBLE_DEVICES=0 python data_gen/tts/bin/binarize.py --config usr/configs/midi/cascade/opencs/aux_rel.yaml

To set up a DiffSinger environment using Anaconda, create a new environment with Python 3.8 and install the requirements specific to your GPU hardware.

For NVIDIA 2080Ti (CUDA 10.2): Use requirements_2080.txt.

For NVIDIA 3090 (CUDA 11.4): Use requirements_3090.txt.

DiffSinger requires a vocoder. A pre-trained HifiGAN-Singing model is provided, which is designed for SVS with the NSF mechanism.

Unzip the downloaded model directly into the checkpoints directory before starting acoustic model training. You can also use a checkpoint with more training steps if available.

You can visualize training progress using Tensorboard by pointing it to your experiment directory.

tensorboard --logdir_spec exp_name

Run the training task using tasks/run.py. You must provide a configuration file, an experiment name, and optionally use --reset to clear previous progress. Use CUDA_VISIBLE_DEVICES to specify the GPU.

export MY_DS_EXP_NAME=0831_opencpop_ds1000
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/midi/e2e/opencpop/ds1000.yaml --exp_name $MY_DS_EXP_NAME --reset