msclap (Contrastive Language-Audio Pretraining)

To use CLAP, ensure you have Python 3.8 or higher installed (Python 3.11 is recommended). You can install the package via PyPI or directly from the GitHub source.

Install via PyPI

pip install msclap

Install latest git source

pip install git+https://github.com/microsoft/CLAP.git

For audio captioning tasks, use the clapcap version of the model. This version uses the 2023 encoders specifically optimized for generating natural language descriptions of audio files.

Workflow:

1. Initialize CLAP with version='clapcap'.
2. Use generate_caption with a list of audio file paths.

from msclap import CLAP

# Load model (Choose version 'clapcap')
clap_model = CLAP(version = 'clapcap', use_cuda=False)

# Generate audio captions
captions = clap_model.generate_caption(file_paths=['path/to/audio.wav'])

The CLAP class is the primary interface for the library.

Initialization

CLAP(version: str, use_cuda: bool = False, model_fp: str = None)

• version: Model version to use. Options: '2022', '2023', or 'clapcap'.
• use_cuda: Whether to use CUDA for acceleration.
• model_fp: Path to model weights. If not provided, weights are downloaded automatically.

Methods

• get_text_embeddings(class_labels: List[str]): Returns embeddings for the provided list of text labels.
• get_audio_embeddings(file_paths: List[str]): Returns embeddings for the provided list of audio file paths.
• compute_similarity(audio_embeddings, text_embeddings): Computes the similarity scores between audio and text embeddings.
• generate_caption(file_paths: List[str]): (Available in 'clapcap' version) Generates text captions for the provided audio files.

Use get_audio_embeddings to convert a list of audio file paths into their corresponding audio embeddings.

Input: A list of file paths (strings). Output: A tensor of audio embeddings.

Note: You can also use get_audio_embeddings_per_batch to process large datasets in chunks to manage memory.

# audio_files is a list of paths
audio_embeddings = wrapper.get_audio_embeddings(audio_files=['path/to/audio1.wav', 'path/to/audio2.wav'])

The CLAPWrapper class is the primary interface for the CLAP model, supporting zero-shot classification, retrieval, and embedding extraction. You can initialize it by specifying a model file path and a version. If model_fp is not provided, the weights will be automatically downloaded from the microsoft/msclap repository on Hugging Face.

Supported versions:

• '2022'
• '2023'
• 'clapcap' (used for caption generation tasks)

from msclap import CLAPWrapper
from pathlib import Path

# Initialize with a specific version (e.g., '2023')
# If model_fp is None, it downloads automatically
wrapper = CLAPWrapper(model_fp=None, version='2023', use_cuda=True)

For large-scale inference, use the per_batch variants of the embedding and classification methods to avoid memory exhaustion.

• get_audio_embeddings_per_batch(audio_files, batch_size): A generator yielding audio embeddings for chunks of files.
• get_text_embeddings_per_batch(class_labels, batch_size): A generator yielding text embeddings for chunks of labels.
• classify_audio_files_per_batch(audio_files, class_labels, batch_size): A generator yielding classification results for batches of audio against all provided labels.

# Example of batch processing audio
for batch_embeddings in wrapper.get_audio_embeddings_per_batch(large_audio_list, batch_size=32):
    # process batch_embeddings
    pass

Use compute_similarity to calculate the similarity scores between audio embeddings and text embeddings. This is typically used for retrieval or zero-shot classification.

Input:

• audio_embeddings: Tensor of audio embeddings.
• text_embeddings: Tensor of text embeddings.

Output: A similarity matrix (tensor) where scores represent the relationship between audio and text pairs.

# audio_embeddings and text_embeddings are obtained via the respective getter methods
similarity_matrix = wrapper.compute_similarity(audio_embeddings, text_embeddings)

The CLAP class (exported from msclap) is the primary entrypoint for interacting with the Contrastive Language-Audio Pretraining model. It provides a high-level wrapper to load model weights, extract text embeddings, extract audio embeddings, and compute similarity between audio and text.

from msclap import CLAP

# Initialize the model
model = CLAP()

Use get_text_embeddings to convert a list of text strings (class labels or queries) into their corresponding text embeddings.

Input: A list of strings. Output: A tensor of text embeddings.

# class_labels is a list of strings
text_embeddings = wrapper.get_text_embeddings(class_labels=['a dog barking', 'a car engine'])