DiffRhythm Documentation

Future updates to DiffRhythm will include support for lyric structure tags (similar to Suno). This will allow users to provide granular control over song sections using tags like:

• [intro]
• [bridge]
• [verse]
• [chorus]

This feature is intended to simplify the process by allowing users to describe song sections rather than manually managing millisecond-based timestamp inputs.

To join the DiffRhythm WeChat group, you must first add a team member manually due to WeChat's group size restrictions.

1. Scan the team member's QR code (contact_person.jpg).
2. Send a request with the specific note: "DiffRhythm Group Invite".
3. Wait for a manual invitation to the group (typically within 24 hours).

Alternatively, you can attempt to scan the Group QR code (wechat_group.jpg) directly.

DiffRhythm supports generating music using literal text descriptions instead of requiring reference audio. You can use various prompting styles:

• Straightforward prompts: Use quick tags for genre and mood (e.g., Pop Emotional Piano or Jazzy Nightclub Vibe).
• Vivid storytelling: Use descriptive, narrative-driven prompts for more complex textures (e.g., Indie folk ballad, coming-of-age themes, acoustic guitar picking with harmonica interludes).

No reference audio is needed when using these text-based style prompts.

To run inference on Windows, you must first configure the espeak-ng environment variables.

1. Install espeak-ng via the .msi installer.
2. Set the following User Environment Variables:
   • PHONEMIZER_ESPEAK_LIBRARY $\rightarrow$ C:\Program Files\eSpeak NG\libespeak-ng.dll
   • PHONEMIZER_ESPEAK_PATH $\rightarrow$ C:\Program Files\eSpeak NG (Note: Adjust the path if you installed eSpeak NG to a different directory.)
3. Reboot your PC to apply the changes.

After setup, use the .bat scripts for inference:

• Reference WAV file: call scripts\infer_wav_ref.bat
• Text prompt reference: call scripts\infer_prompt_ref.bat

Note: English lyrics will be used by default on Windows.

You can generate instrumental music by providing prompts and omitting lyrics. This mode allows for highly specific or even abstract prompts to define the musical atmosphere.

Examples of instrumental prompts:

• Mountain cabin fireplace, acoustic guitar folk tunes wrapped in wool blankets
• Arctic research station, theremin auroras dancing with geomagnetic storms
• Zombie apocalypse country-rock gas station escape, banjo shreds & shotgun reload beats

Instrumental mode is ideal for creating background textures or experimental sounds where vocal content is not desired.

To deploy DiffRhythm using Docker, use the configuration files located in the docker directory.

1. Clone the repository or copy the docker folder contents.
2. Navigate into the folder.
3. Edit the docker compose binding folders to match your local paths.
4. Start the containers using docker compose up -d.
5. Enter the running container using docker exec -it DiffRhythm bash.
6. Once inside, navigate to /home/app/scripts to run inference scripts like infer_prompt_ref.sh.

Once the environment is set up, you can run inference using provided shell scripts.

• Reference WAV file: Use scripts/infer_wav_ref.sh to generate music based on an existing audio file.
• Text prompt reference: Use scripts/infer_prompt_ref.sh to generate music based on a text description.

VRAM Optimization: If you are using DiffRhythm-base and have limited VRAM (minimum 8G required), use the --chunked argument during inference to enable chunked decoding.

For users who do not use WeChat, you can join the DiffRhythm community for discussions via our Discord server.

https://discord.gg/vUD4zgTpJa

You can run the DiffRhythm environment using Docker Compose. The configuration is set up to build an image from the local Dockerfile and requires an NVIDIA GPU with the nvidia driver to function correctly, as it reserves one GPU device for the container.

By default, the container is configured to run a long-lived bash process (tail -f /dev/null) with stdin_open and tty enabled, allowing you to attach to the container and run commands interactively.

If the WeChat contact or group QR codes provided in the documentation expire, please open an issue on GitHub so the team can refresh them.

https://github.com/ASLP-lab/DiffRhythm/issues

The docker-compose.yml file includes a deploy section that reserves hardware resources. To ensure the container can access the GPU for inference, the following configuration is used:

• driver: nvidia
• count: 1
• capabilities: [gpu]

Ensure you have the NVIDIA Container Toolkit installed on your host machine for these reservations to work.

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: 1
          capabilities: [gpu]