DOCSEARCHbyAlgolia
Submit Docs

HeartMuLa

repository·main·Indexed 25 days ago

https://github.com/heartmula/heartlib

A Python library for HeartMuLa, a family of open-source music foundation models. It provides tools for high-fidelity music generation (HeartMuLa), music codec operations (HeartCodec), lyrics transcription (HeartTranscriptor), and audio-text alignment (HeartCLAP). The library includes pipelines like HeartMuLaGenPipeline and HeartTranscriptorPipeline for end-to-end music generation and transcription tasks.

Tokens
3.4K
Snippets
3
Records
23
Agent score
87%

What's inside heartlib

  1. Download HeartMuLa pretrained checkpoints

    main

    You can download the required pretrained checkpoints from either Hugging Face or ModelScope. The checkpoints should be organized into a ./ckpt directory.

    Using Hugging Face

    hf download --local-dir './ckpt' 'HeartMuLa/HeartMuLaGen'
hf download --local-dir './ckpt/HeartMuLa-oss-3B' 'HeartMuLa/HeartMuLa-oss-3B-happy-new-year'
hf download --local-dir './ckpt/HeartCodec-oss' HeartMuLa/HeartCodec-oss-20260123

    Using ModelScope

    modelscope download --model 'HeartMuLa/HeartMuLaGen' --local_dir './ckpt'
modelscope download --model 'HeartMuLa/HeartMuLa-oss-3B-happy-new-year' --local_dir './ckpt/HeartMuLa-oss-3B'
modelscope download --model 'HeartMuLa/HeartCodec-oss-20260123' --local_dir './ckpt/HeartCodec-oss'

    Expected directory structure:

    ./ckpt/
├── HeartCodec-oss/
├── HeartMuLa-oss-3B/
├── gen_config.json
└── tokenizer.json
    # if you are using huggingface
hf download --local-dir './ckpt' 'HeartMuLa/HeartMuLaGen'
hf download --local-dir './ckpt/HeartMuLa-oss-3B' 'HeartMuLa/HeartMuLa-oss-3B-happy-new-year'
hf download --local-dir './ckpt/HeartCodec-oss' HeartMuLa/HeartCodec-oss-20260123


# if you are using modelscope
modelscope download --model 'HeartMuLa/HeartMuLaGen' --local_dir './ckpt'
modelscope download --model 'HeartMuLa/HeartMuLa-oss-3B-happy-new-year' --local_dir './ckpt/HeartMuLa-oss-3B'
modelscope download --model 'HeartMuLa/HeartCodec-oss-20260123' --local_dir './ckpt/HeartCodec-oss'

  2. Download the HeartTranscriptor-oss checkpoint

    main
    You can download the HeartTranscriptor-oss model checkpoint using either Hugging Face or ModelScope. Save the checkpoint to a local directory (e.g., ./ckpt/HeartTranscriptor-oss) to use it with the transcription script.

  4. Manage VRAM with lazy_load in HeartMuLaGenPipeline

    main

    If you initialize the pipeline with lazy_load=True, the pipeline will attempt to manage GPU memory by unloading models after use.

    • Loading: Models (HeartMuLa and HeartCodec) are loaded onto the specified devices only when the .mula or .codec properties are first accessed during the generation process.
    • Unloading: After the _forward and postprocess steps are complete, the pipeline calls an internal _unload() method which deletes the model objects, triggers garbage collection (gc.collect()), and clears the CUDA cache (torch.cuda.empty_cache()).

    Note: If you specify different devices for mula and codec, lazy_load is automatically disabled to ensure stability.

  5. Manage GPU memory for HeartMuLa

    main

    If you encounter CUDA out-of-memory errors, use the following strategies:

    1. Multi-GPU Setup: Place HeartMuLa and HeartCodec on different devices using --mula_device and --codec_device.
      --mula_device cuda:0 --codec_device cuda:1
    2. Single GPU Setup: Enable lazy loading to load modules on demand and delete them after inference.
      --lazy_load true

  6. Generate music with HeartMuLa

    main

    Run the run_music_generation.py script to generate music conditioned on lyrics and tags. By default, it uses files from the ./assets folder and saves the output to ./assets/output.mp3.

    python ./examples/run_music_generation.py --model_path=./ckpt --version="3B"

  7. Run lyrics transcription with HeartTranscriptor

    main

    Execute the transcription script using run_lyrics_transcription.py.

    By default, the script loads the music file located at ./assets/output.mp3 and prints the transcribed lyrics to the console.

    Important Note on Accuracy: HeartTranscriptor is trained on separated vocal tracks. While the example works on unseparated music tracks for simplicity, you should use source separation tools like demucs to separate the vocals from the music before transcription to achieve optimal results.

  9. Configure HeartMuLa backbone and decoder flavors

    main

    The model architecture is determined by the backbone_flavor and decoder_flavor keys in the HeartMuLaConfig. The following flavors are supported:

    • llama-3B: 3B parameter Llama 3.2 variant
    • llama-300M: 300M parameter Llama 3.2 variant
    • llama-7B: 7B parameter Llama 3.2 variant
    • llama-400M: 400M parameter Llama 3.2 variant (optimized for precision over efficiency)

  10. Configure HeartCodecConfig for HeartCodec models

    main

    The HeartCodecConfig class is used to define the configuration for the HeartCodec model. It inherits from PretrainedConfig and is divided into three functional groups: RVQ (Residual Vector Quantization), Diffusion Transformer, and SQ Codec.

    RVQ Configuration

    • dim (int, default: 512): Embedding dimension.
    • codebook_size (int, default: 8192): Size of the codebook.
    • decay (float, default: 0.9): Decay factor.
    • commitment_weight (float, default: 1.0): Weight for commitment loss.
    • threshold_ema_dead_code (int, default: 2): Threshold for EMA dead code handling.
    • use_cosine_sim (bool, default: False): Whether to use cosine similarity.
    • codebook_dim (int, default: 32): Dimension of the codebook.
    • num_quantizers (int, default: 8): Number of quantizers.

    Diffusion Transformer Configuration

    • attention_head_dim (int, default: 64): Dimension of attention heads.
    • in_channels (int, default: 1024): Number of input channels.
    • norm_type (str, default: `

  11. Configure music generation parameters

    main

    The run_music_generation.py script accepts several arguments to control the generation process:

    ArgumentDescription
    --model_path (required)Path to the pretrained model checkpoint
    --lyricsPath to lyrics file (default: ./assets/lyrics.txt)
    --tagsPath to tags file (default: ./assets/tags.txt)
    --save_pathOutput audio file path (default: ./assets/output.mp3)
    --max_audio_length_msMaximum audio length in milliseconds (default: 240000)
    --topkTop-k sampling parameter (default: 50)
    --temperatureSampling temperature (default: 1.0)
    --cfg_scaleClassifier-free guidance scale (default: 1.5)
    --versionModel version: 3B or 7B (default: 3B)
    --mula_deviceDevice for HeartMuLa params (default: cuda)
    --codec_deviceDevice for HeartCodec params (default: cuda)
    --mula_dtypeInference dtype for HeartMuLa (default: bf16)
    --codec_dtypeInference dtype for HeartCodec (default: fp32)
    --lazy_loadUse lazy loading to save GPU memory (default: false)

    Input Formats

    Lyrics: Use structural tags like [Intro], [Verse], [Chorus], etc.

    Tags: Comma-separated values without spaces (e.g., piano,happy,wedding,synthesizer,romantic).

  12. Initialize the HeartMuLa model

    main
    The HeartMuLa class is a PreTrainedModel used for music-related generative tasks. It requires a HeartMuLaConfig object during initialization. The model architecture is composed of a backbone and a decoder, which are instantiated based on the backbone_flavor and decoder_flavor specified in the configuration. Available flavors include llama-3B, llama-300M, llama-7B, and llama-400M.