DOCSEARCHbyAlgolia
Submit Docs

CLaMP 3

repository·main·Indexed 42 days ago

https://github.com/sanderwood/clamp3

A framework for multimodal and multilingual music information retrieval (MIR) that aligns text, sheet music, audio, MIDI, and images into a shared representation space using contrastive learning. It includes tools for feature extraction via MERT and M3 encoders, semantic retrieval, similarity scoring, and linear classification training.

Tokens
5.4K
Snippets
27
Records
37
Agent score
53%

What's inside CLaMP 3

  1. Overview of CLaMP 3 preprocessing capabilities

    main
    The preprocessing/ directory provides a suite of scripts and utilities for converting and processing musical data across three primary modalities: ABC Notation, Audio, and MIDI. These tools are designed to standardize data formats and extract features required for downstream multimodal music information retrieval tasks.

  2. CLaMP 3 Repository Structure Overview

    main

    The repository is organized into the following functional areas:

    • code/: Contains training and feature extraction scripts. Note: Model weights should be placed in this folder.
    • classification/: Contains scripts for linear classification training and prediction.
    • inference/: Contains scripts for semantic search, similarity calculations, and retrieval evaluation.
    • preprocessing/: Contains tools to convert data into Interleaved ABC, MTF, or MERT-extracted features.

  4. Prepare feature files for classification

    main

    To use the classification codebase, your feature files must adhere to the following requirements:

    Data Shape & Dimensionality:

    • Features must be single, fixed-length vectors (e.g., shape (1, 768) or (768)). Sequential or time-series data is not supported.
    • All feature files in a set must have the same dimensionality.

    Naming Convention: Files must be .npy files named using the following pattern to allow label extraction: label_filename.npy

    Important: The filename should not contain any underscores (_) except for the single underscore separating the label from the filename.

  7. Workflow for CLaMP 3 ABC preprocessing

    main

    To prepare music data for CLaMP 3, follow this two-step pipeline:

    1. Convert MusicXML to Standard ABC: python batch_xml2abc.py <input_dir> <output_dir>
    2. Convert Standard ABC to Interleaved ABC: python batch_interleaved_abc.py <input_dir> <output_dir>

    The resulting Interleaved ABC files are the format required by the model.

  8. Use pre-trained CLaMP 3 models

    main

    CLaMP 3 provides two optimized versions. By default, the system is configured for the SAAS version (audio-optimized).

    • CLaMP 3 SAAS: Best for audio-based retrieval.
    • CLaMP 3 C2: Best for symbolic music retrieval (MIDI, sheet music).

    To switch from the default SAAS version to the C2 version, modify line 66 in config.py by changing the value from "saas" to "c2".

  10. Train CLaMP 3 on symbolic or audio data

    main

    While using pre-trained weights is recommended, you can train CLaMP 3 on your own data. You must first modify config.py to adjust hyperparameters and data paths.

    • Symbolic Music (Sheet music, MIDI): Use train_clamp3_symbolic.py.
    • Audio Data: Use train_clamp3_audio.py.
    # Training on symbolic music
python -m torch.distributed.launch --nproc_per_node=<GPUs> --use_env train_clamp3_symbolic.py

# Training on audio data
python -m torch.distributed.launch --nproc_per_node=<GPUs> --use_env train_clamp3_audio.py

  11. Extract CLaMP 3 features

    main

    Use extract_clamp3.py to extract features from your processed data. These are stored as .npy files.

    For retrieval tasks: You must include the --get_global flag to extract a global semantic vector for each input. If you omit this flag, the model will not work correctly for retrieval (it is only omitted for downstream fine-tuning or custom raw feature extraction).

    accelerate launch extract_clamp3.py --epoch <epoch> <input_dir> <output_dir> --get_global

  12. Train the M3 symbolic music encoder

    main

    The train_m3.py script trains the M3 model, which encodes interleaved ABC and MTF files. You can specify training and evaluation directories using the TRAIN_FOLDERS and EVAL_FOLDERS variables in the configuration.

    Note: Retraining M3 is generally unnecessary for most users as the pre-trained M3 model is typically sufficient.

    python -m torch.distributed.launch --nproc_per_node=<number_of_GPUs> --use_env train_m3.py