DOCSEARCHbyAlgolia
Submit Docs

YuE Music Generation Model

repository·main·Indexed 27 days ago

https://github.com/multimodal-art-projection/yue

An open-source foundation model series for lyrics-to-song generation, capable of producing full songs with vocals and accompaniment across various genres and languages. It supports Chain-of-Thought (CoT) and In-Context Learning (ICL) inference modes, as well as LoRA finetuning. The project includes tools for pitch extraction using RMVPE and data preprocessing for Megatron binary format.

Tokens
5K
Snippets
9
Records
23
Agent score
92%

What's inside YuE

  1. Overview of YuE (乐)

    main
    YuE is an open-source foundation model series designed for music generation, specifically the lyrics2song task. It transforms lyrics into complete songs containing both vocal and accompaniment tracks. It supports diverse genres, languages, and vocal techniques. The models are licensed under Apache 2.0, allowing for sampling and monetization of outputs provided that credit is given to YuE by HKUST/M-A-P.

  2. Hardware Requirements and Performance Considerations

    main

    When running YuE, pay attention to the following hardware constraints to ensure stability:

    • GPU Memory: To prevent Out-of-Memory (OOM) errors, especially when processing long audio, installing FlashAttention 2 is mandatory.
    • CUDA: Ensure you have CUDA >= 11.8 installed.

  4. Quickstart: Install environment and dependencies

    main

    To set up the YuE environment, follow these recommendations:

    1. Environment Management: It is recommended to use conda to create a new, isolated environment.
    2. CUDA: Install CUDA >= 11.8.
    3. Memory Optimization: Install FlashAttention 2 to save GPU memory and avoid OOM errors during long audio generation.

  5. Download inference code and tokenizer

    main

    You need git-lfs installed to clone the repository and the tokenizer. Follow these steps to download the inference directory and the xcodec_mini_infer tokenizer from Hugging Face.

    # Install git-lfs
sudo apt update
sudo apt install git-lfs
git lfs install

# Clone the main repository
git clone https://github.com/multimodal-art-projection/YuE.git

# Download tokenizer into the inference directory
cd YuE/inference/
git clone https://huggingface.co/m-a-p/xcodec_mini_infer

  7. Install the YuE finetuning environment

    main

    To set up the environment for finetuning YuE, it is recommended to use Python 3.10, PyTorch 2.4, and CUDA 12.1+. Use the following commands to clone the repository and create a dedicated conda environment:

    git clone https://github.com/multimodal-art-projection/YuE.git
cd YuE/finetune/
conda create -n yue-ft python=3.10
conda activate yue-ft
pip install -r requirements.txt
    git clone https://github.com/multimodal-art-projection/YuE.git
cd YuE/finetune/
conda create -n yue-ft python=3.10
conda activate yue-ft
pip install -r requirements.txt

  9. Prepare YuE finetuning data structure and JSONL format

    main

    Data must be organized into a specific directory structure and follow a strict JSONL schema for the preprocessing scripts to work.

    Directory Structure

    example/
├── jsonl/     # Source JSONL files
├── mmap/      # Generated Megatron binary files
└── npy/       # Discrete audio codes (numpy arrays) from xcodec

    JSONL Entry Schema

    Each line in your JSONL file must be a JSON object containing:

    • id: Unique identifier.
    • codec: Path to raw audio codes (.npy).
    • vocals_codec: Path to vocal track codes (.npy).
    • instrumental_codec: Path to instrumental track codes (.npy).
    • audio_length_in_sec: Duration in seconds.
    • msa: Music Structure Analysis array containing start, end, and label (e.g., intro).
    • genres: Comma-separated tags for gender, age, genre, mood, and timbre.
    • splitted_lyrics: Object containing segmented_lyrics with offset, duration, codec_frame_start, codec_frame_end, and line_content.

  10. Configure Genre Tagging Prompts

    main

    Genre tags guide the musical style. A stable prompt should ideally include five components separated by spaces: genre, instrument, mood, gender, and timbre.

    Example: inspiring female uplifting pop airy vocal electronic bright vocal vocal.

    • Use tags from the provided top_200_tags.json for more stable results.
    • Use Mandarin or Cantonese tags to distinguish between those languages.
    • Tag order is flexible.

  11. Configure Audio Prompts for ICL

    main

    Audio prompts are used for In-Context Learning (ICL) to increase prompt following and musicality, though they may reduce output diversity compared to CoT mode.

    Best Practices:

    • Duration: Approximately 30s of audio is recommended.
    • Content: Using the chorus part of a song as a prompt typically results in better musicality.
    • Mode: Dual-track ICL (vocal + instrumental) is generally superior to single-track ICL.