repository·main·Indexed 27 days ago
https://github.com/facebookresearch/imagebindA PyTorch implementation of a multimodal joint embedding model by Meta AI. ImageBind learns a unified embedding space across six modalities: images/video, text, audio, depth, thermal, and IMU data, enabling cross-modal retrieval and zero-shot classification. The model utilizes an OpenCLIP ViT-H encoder for image and text and is intended strictly for research purposes.
ImageBind is a multimodal joint embedding model developed by Meta AI. It provides a single, unified embedding space for six different modalities: image/video, text, audio, depth, IMU, and thermal images.
Developers can input any of these modalities to obtain same-sized embeddings, enabling cross-modal tasks such as cross-modal retrieval or combining embeddings from different modalities for research purposes.
To install ImageBind, you need PyTorch 2.0+ and other 3rd party dependencies. It is recommended to use a Conda environment. Windows users should also install soundfile to handle audio files.
conda create --name imagebind python=3.10 -y
conda activate imagebind
pip install .For Windows:
pip install soundfileImageBind allows you to extract embeddings for multiple modalities (Vision, Text, Audio, etc.) and compare them using dot products. You can use imagebind_model.imagebind_huge(pretrained=True) to instantiate the model and the data module to load and transform inputs into the required format for the model.
Supported modalities via ModalityType include TEXT, VISION, and AUDIO.
from imagebind import data
import torch
from imagebind.models import imagebind_model
from imagebind.models.imagebind_model import ModalityType
text_list=["A dog.", "A car", "A bird"]
image_paths=[".assets/dog_image.jpg", ".assets/car_image.jpg", ".assets/bird_image.jpg"]
audio_paths=[".assets/dog_audio.wav", ".assets/car_audio.wav", ".assets/bird_audio.wav"]
device = "cuda:0" if torch.cuda.is_available() else "cpu"
# Instantiate model
model = imagebind_model.imagebind_huge(pretrained=True)
model.eval()
model.to(device)
# Load data
inputs = {
ModalityType.TEXT: data.load_and_transform_text(text_list, device),
ModalityType.VISION: data.load_and_transform_vision_data(image_paths, device),
ModalityType.AUDIO: data.load_and_transform_audio_data(audio_paths, device),
}
with torch.no_grad():
embeddings = model(inputs)
print(
"Vision x Text: ",
torch.softmax(embeddings[ModalityType.VISION] @ embeddings[ModalityType.TEXT].T, dim=-1),
)
print(
"Audio x Text: ",
torch.softmax(embeddings[ModalityType.AUDIO] @ embeddings[ModalityType.TEXT].T, dim=-1),
)
print(
"Vision x Audio: ",
torch.softmax(embeddings[ModalityType.VISION] @ embeddings[ModalityType.AUDIO].T, dim=-1),
)ImageBind uses image-paired data (image, X) where X is one of the following modalities. The model initializes and freezes image and text encoders using an OpenCLIP ViT-H encoder.
| Modality | Training Dataset |
|---|---|
| Audio | Audioset |
| Depth | SUN RGB-D |
| IMU | Ego4D |
| Thermal | LLVIP |
| Text | (Uses OpenCLIP ViT-H) |
The imagebind_huge model checkpoint can be downloaded from the following URL: