LanguageBind

To train LanguageBind on a specific modality (e.g., Depth-Language), follow these steps:

1. Download Pretrained Weights: Download the appropriate cache (Large or Huge) from the provided links (Baidu Yun, Google Cloud, or Peking University Yun).
2. Set Cache Directory: Define the CACHE_DIR environment variable to the path where you stored the weights.
3. Prepare Data: Set the ANNOTATION path to your prepared data directory. Ensure your dataset follows the required structure (see Downstream datasets).
4. Execute Training: Run the training script using torchrun.

Note: The example below uses 8 GPUs on 1 node. Adjust --nproc_per_node and --nnodes as needed.

CACHE_DIR="/path/to/LanguageBind"
ANNOTATION="path/to/data"
cd /path/to/LanguageBind
TORCH_DISTRIBUTED_DEBUG=DETAIL HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 torchrun --nnodes=1 --nproc_per_node 8 \
    -m main  \
    --train-data ${ANNOTATION} \
    --train-num-samples 3020000 \
    --clip-type "dl" --max-depth 10 \
    --do_train \
    --lock-text --lock-image --text-type "polish_mplug" \
    --init-temp 0.07 --learn-temp \
    --model "ViT-L-14" --cache-dir ${CACHE_DIR} \
    --convert_to_lora --lora_r 2 \
    --lr 5e-4 --coef-lr 1e-3 \
    --beta1 0.9 --beta2 0.98 --wd 0.2 --eps 1e-6 \
    --num-frames 1 --force-patch-dropout 0.5 \
    --epochs 1 --batch-size 128 --accum-freq 1 --warmup 200 \
    --precision "amp" --workers 10 --video-decode-backend "imgs" \
    --save-frequency 1 --log-every-n-steps 20 --report-to "tensorboard" --resume "latest" \
    --do_eval \
    --val_d_cls_data "NYUV2"

You can use the LanguageBind class to perform multi-modal binding across various modalities including video, audio, thermal, image, and depth. This approach allows you to compute embeddings for multiple modalities simultaneously and perform zero-shot comparisons (e.g., Video x Text, Image x Audio) using the shared semantic space.

To use this, you need to:

1. Define a clip_type dictionary mapping modality names to their respective model identifiers.
2. Initialize the LanguageBind model and a LanguageBindImageTokenizer.
3. Create a modality_transform dictionary using transform_dict and the model's modality_config.
4. Prepare input lists for each modality and tokenize the text.
5. Pass the dictionary of transformed inputs to the model to get embeddings.

import torch
from languagebind import LanguageBind, to_device, transform_dict, LanguageBindImageTokenizer

if __name__ == '__ '__:
    device = torch.device('cuda:0')
    clip_type = {
        'video': 'LanguageBind_Video_FT',
        'audio': 'LanguageBind_Audio_FT',
        'thermal': 'LanguageBind_Thermal',
        'image': 'LanguageBind_Image',
        'depth': 'LanguageBind_Depth',
    }

    model = LanguageBind(clip_type=clip_type, cache_dir='./cache_dir')
    model = model.to(device)
    model.eval()
    
    pretrained_ckpt = 'lb203/LanguageBind_Image'
    tokenizer = LanguageBindImageTokenizer.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir/tokenizer_cache_dir')
    modality_transform = {c: transform_dict[c](model.modality_config[c]) for c in clip_type.keys()}

    # Example inputs
    image = ['assets/image/0.jpg']
    language = ["Training a parakeet to climb up a ladder."]

    inputs = {
        'image': to_device(modality_transform['image'](image), device),
        'language': to_device(tokenizer(language, max_length=77, padding='max_length', truncation=True, return_tensors='pt'), device),
    }

    with torch.no_grad():
        embeddings = model(inputs)

    # Compute similarity
    print(torch.softmax(embeddings['image'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())

LanguageBind supports several modalities. Most datasets are reformatted to conform to the standard ImageNet format. When using custom data, you must update the data_root in data/build_datasets.py for the specific modality.

Modalities and Sources:

• Depth: Uses NYU V2. Reformat to ImageNet style.
• Video: Downloaded from HBI repository.
• Audio: Uses ONE-PEACE and AudioSet. Reformat to ImageNet style.
• Infrared (Thermal): Uses LLVIP and FLIR. Reformat to ImageNet style.

Required Folder Structure:

Datasets should be organized under a downstream_datasets root following this pattern:

downstream_datasets
├── Audio
│   ├── audiocaps
│   │   └── audio
│   │       ├── test
│   │       ├── train
│   │       └── val
│   ├── audioset
│   │   └── ...
│   └── ...
├── Depth
│   └── nyuv2
│       └── data
│           └── val
│               └── [class_folders]
├── Thermal
│   ├── flirv1
│   ├── flirv2
│   └── llvip
└── VideoTextRetrieval
    └── vtRetdata
        └── [dataset_name]
            └── Videos

downstream_datasets
├── Audio
│   ├── audiocaps
│   │   └── audio
│   │       ├── test
│   │       ├── train
│   │       └── val
│   ├── audioset
│   │   ├── balanced_train_segments
│   │   ├── eval_segments
│   │   └── unbalanced_train_segments
│   │       ├── unbalanced_train_segments_part00
│   │       ├── unbalanced_train_segments_part01
│   │       └── ...
│   ├── clotho
│   │   ├── CLOTHO_retrieval_dataset
│   │   └── evaluation
│   └── esc50
│       └── test
│           ├── airplane
│           ├── breathing
│           └── ...
├── laionaudio
│   ├── audios
│   ├── freesound_no_overlap
│   └── jsons
├── vggsound
│       └── test
│           └── ...
├── Depth
│   ├── nyuv2
│   │   └── data
│   │       └── val
│   │          ├── bathroom
│   │          ├── bedroom
│   │          └── ...
├── Thermal
│   ├── flirv1
│   │   └── val
│   │       └── ...
│   ├── flirv2
│   │   └── val
│   │       └── ...
│   ├── llvip
│   │   ├── train
│   │   └── val
└── VideoTextRetrieval
    ├── vtRetdata
    │   ├── ActivityNet
    │   │   └── Videos
    │   ├── Didemo
    │   │   └── videos
    │   ├── MSRVTT
    │   │   └── MSRVTT_Videos
    │   └── MSVD
    │       └── MSVD_Videos

To validate a trained LanguageBind model (e.g., on Depth-Language) using 1 GPU:

1. Specify Checkpoint: Set the RESUME variable to the path of your model checkpoint (e.g., thermal_language.pt).
2. Prepare Downstream Dataset: Ensure the validation dataset is prepared according to the project requirements.
3. Execute Validation: Run the script using torchrun with --nproc_per_node 1 and the --resume ${RESUME} flag.

CACHE_DIR="/path/to/LanguageBind"
RESUME="thermal_language.pt"
ANNOTATION="path/to/data"
cd /path/to/LanguageBind
TORCH_DISTRIBUTED_DEBUG=DETAIL HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 torchrun --nproc_per_node 1 \
    -m main  \
    --train-data ${ANNOTATION} \
    --train-num-samples 3020000 \
    --clip-type "dl" --max-depth 10 \
    --lock-text --lock-image --text-type "polish_mplug" \
    --init-temp 0.07 --learn-temp \
    --model "ViT-L-14" --cache-dir ${CACHE_DIR} \
    --convert_to_lora --lora_r 2 \
    --lr 5e-4 --coef-lr 1e-3 \
    --beta1 0.9 --beta2 0.98 --wd 0.2 --eps 1e-6 \
    --num-frames 1 --force-patch-dropout 0.5 \
    --epochs 1 --batch-size 128 --accum-freq 1 --warmup 200 \
    --precision "amp" --workers 10 --video-decode-backend "imgs" \
    --save-frequency 1 --log-every-n-steps 20 --report-to "tensorboard" --resume ${RESUME} \
    --do_eval \
    --val_d_cls_data "NYUV2"