DeepFuze

The project provides experimental implementations of several vocoder models that can be combined with other TTS models. The currently implemented models are:

• Melgan
• MultiBand-Melgan
• ParallelWaveGAN
• GAN-TTS (Discriminator Only)

To train a new vocoder model, follow these steps:

1. Gather all .wav files into a single folder.
2. Configure a config.json file. You must set the data_path key to the path of your wav files folder and define other relevant training parameters.
3. Example configuration files are available in the tts/vocoder/configs/ directory.
4. Execute the training script using the --config_path flag.

CUDA_VISIBLE_DEVICES='0' python tts/bin/train_vocoder.py --config_path path/to/config.json

To generate embedding vectors from a dataset, use the compute_embeddings.py script. This script parses all .wav files at the specified dataset path and recreates the same folder structure under the provided output path, containing the generated embedding files.

Requirements:

• A pretrained model (downloadable from Mozilla TTS Released Models).
• A configuration file (config.json).

python speaker_encoder/compute_embeddings.py --use_cuda true /model/path/best_model.pth model/config/path/config.json dataset/path/ output_path

To enable significant speed increases on Windows, you must install NVIDIA CUDA and cuDNN.

Requirements:

• NVIDIA CUDA Toolkit: Version 11.8.
• cuDNN: Version 8.9.2.26 (Download from NVIDIA Developer Archive or provided Google Drive link).

Steps:

1. Install CUDA 11.8 and cuDNN 8.9.2.26.
2. Add the installation paths to your system Environment Variables.
3. Restart your computer.
4. Verify installation by running nvcc --version in your terminal. It should return a response confirming release 11.8.

Troubleshooting zlibwapi.dll error: If you see a missing zlibwapi.dll error, download the NVIDIA zlibwapi.dll file and copy it to: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\zlibwapi.dll

To install DeepFuze on Windows using Portable ComfyUI, follow these steps:

1. Install Build Tools: Install Visual Studio (Community version is fine) or VS C++ Build Tools with the "Desktop Development with C++" workload selected.
2. Install via ComfyUI-Manager:
   • Search for DeepFuze in the Manager and install it.
   • Restart ComfyUI.
   • Alternatively, use "Install Via GIT URL" in the Manager and paste: https://github.com/SamKhoze/ComfyUI-DeepFuze.git.
3. Restart ComfyUI.

Note: If you are unsure about CUDA installation, use the CPU version to avoid complexity.

Do not use ComfyUI-Manager on Mac. You must install manually:

1. Set Environment Variable: Before running, execute: export PYTORCH_ENABLE_MPS_FALLBACK=1
2. Install Dependencies:
   • Install PyTorch (tested on M1/M3).
   • pip install onnxruntime (Note: Use onnxruntime, NOT onnxruntime-gpu).
   • pip install dlib.
   • pip install TTS (for Voice Cloning).
3. Clone Repository:

   cd custom_nodes
   git clone https://github.com/SamKhoze/CompfyUI-DeepFuze.git
   cd CompfyUI-DeepFuze
   pip install -r requirements.txt

4. Required Extra Nodes: You must also clone ComfyUI-VideoHelperSuite into your custom_nodes folder to load video and audio.

To fine-tune an existing model, use the --restore_path flag pointing to a .pth checkpoint file.

Note on behavior: Restoring a model via --restore_path starts a new training run in a different folder and only restores the weights from the checkpoint. This differs from --continue_path, which resumes training in the original directory.

CUDA_VISIBLE_DEVICES='0' python tts/bin/train_vocoder.py --restore_path path/to/your/model.pth

You can train the speaker encoder by following the standard TTS workflow. You must first define a config.json file tailored to your needs; ensure that audio parameters in the config match your TTS model.

To initiate training, use the train.py script with the --config_path and --data_path flags. Training progress can be monitored using Tensorboard.

python speaker_encoder/train.py --config_path speaker_encoder/config.json --data_path ~/Data/Libri-TTS/train-clean-360

Models must be manually downloaded and placed in the following directory: ./ComfyUI/models/deepfuze/

Models can be found on Google Drive. Ensure you download each model individually due to their large size.

If you want to resume training from where a previous run left off (starting in the same directory), use the --continue_path flag pointing to your model folder.

CUDA_VISIBLE_DEVICES='0' python tts/bin/train_vocoder.py --continue_path path/to/your/model/folder