CLIP4Clip

repository·master·Indexed 21 days ago

A video-text retrieval model based on OpenAI's CLIP. It implements various similarity calculation approaches—parameter-free, sequential, and tight types—to achieve state-of-the-art results on video retrieval benchmarks including MSRVTT, MSVD, ActivityNet, and DiDeMo.

Tokens: 2.1K
Snippets: 8
Records: 8
Agent score: 27%

What's inside CLIP4Clip

Install CLIP4Clip requirements
master
To set up the environment for CLIP4Clip, you need to install PyTorch, torchvision, and several Python libraries. The setup assumes a Conda environment for the core deep learning stack.
Run the following commands to install the necessary dependencies:
```
# From CLIP
conda install --yes -c pytorch pytorch=1.7.1 torchvision cudatoolkit=11.0
pip install ftfy regex tqdm
pip install opencv-python boto3 requests pandas
```
https://github.com/arrowluo/clip4clip/blob/master/README.md
Prepare data for MSVD
master
To use the MSVD dataset, download the raw videos from the official source and the splits/captions from the CLIP4Clip releases.
1. Download splits and raw_captions:
```
wget https://github.com/ArrowLuo/CLIP4Clip/releases/download/v0.0/msvd_data.zip
```
https://github.com/arrowluo/clip4clip/blob/master/README.md
Prepare data for MSRVTT
master
To use the MSRVTT dataset, you need the official data/video links and the splits/captions. You can download the splits and captions directly from the CLIP4Clip releases, and the raw videos from the Frozen in Time repository.
1. Download splits and captions:
```
wget https://github.com/ArrowLuo/CLIP4Clip/releases/download/v0.0/msrvtt_data.zip
```
1. Download raw videos:
```
wget https://www.robots.ox.ac.uk/~maxbain/frozen-in-time/data/MSRVTT.zip
```
```
wget https://github.com/ArrowLuo/CLIP4Clip/releases/download/v0.0/msrvtt_data.zip

wget https://www.robots.ox.ac.uk/~maxbain/frozen-in-time/data/MSRVTT.zip
```
https://github.com/arrowluo/clip4clip/blob/master/README.md

Download CLIP pretrained weights

master

CLIP4Clip requires pretrained CLIP weights. You can download either the ViT-B/32 or ViT-B/16 models into the ./modules directory.

# Download CLIP (ViT-B/32)
wget -P ./modules https://openaipublic.azureedge.net/clip/models/40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af/ViT-B-32.pt

# Download CLIP (ViT-B/16)
wget -P ./modules https://openaipublic.azureedge.net/clip/models/5806e77cd80f8b59890b7e101eabd078d9fb84e6937f9e85e4ecb61988df416f/ViT-B-16.pt

Compress videos for speed-up
master
You can optionally compress videos to 3fps with a width or height of 224 to speed up processing. Use the preprocess/compress_video.py script for this purpose.
```
python preprocess/compress_video.py --input_root [raw_video_path] --output_root [compressed_video_path]
```
https://github.com/arrowluo/clip4clip/blob/master/README.md

Run training for ActivityNet or DiDeMo

master

ActivityNet and DiDeMo are treated as video-paragraph retrieval tasks. These require more GPUs (e.g., 8 GPUs) or multi-node execution due to the larger scale of the data.

# ActivityNet Example
DATA_PATH=[Your ActivityNet data and videos path]
python -m torch.distributed.launch --nproc_per_node=8 \
main_task_retrieval.py --do_train --num_thread_reader=2 \
--epochs=5 --batch_size=128 --n_display=50 \
--data_path ${DATA_PATH} \
--features_path ${DATA_PATH}/Activity_Videos \
--output_dir ckpts/ckpt_activity_retrieval_looseType \
--lr 1e-4 --max_words 64 --max_frames 64 --batch_size_val 16 \
--datatype activity --feature_framerate 1 --coef_lr 1e-3 \
--freeze_layer_num 0 --slice_framepos 2 \
--loose_type --linear_patch 2d --sim_header meanP \
--pretrained_clip_name ViT-B/32

Run training for MSRVTT

master

Execute the training pipeline for the MSRVTT dataset using distributed training. Ensure DATA_PATH is set to your local data and video directory.

DATA_PATH=[Your MSRVTT data and videos path]
python -m torch.distributed.launch --nproc_per_node=4 \
main_task_retrieval.py --do_train --num_thread_reader=0 \
--epochs=5 --batch_size=128 --n_display=50 \
--train_csv ${DATA_PATH}/MSRVTT_train.9k.csv \
--val_csv ${DATA_PATH}/MSRVTT_JSFUSION_test.csv \
--data_path ${DATA_PATH}/MSRVTT_data.json \
--features_path ${DATA_PATH}/MSRVTT_Videos \
--output_dir ckpts/ckpt_msrvtt_retrieval_looseType \
--lr 1e-4 --max_words 32 --max_frames 12 --batch_size_val 16 \
--datatype msrvtt --expand_msrvtt_sentences  \
--feature_framerate 1 --coef_lr 1e-3 \
--freeze_layer_num 0  --slice_framepos 2 \
--loose_type --linear_patch 2d --sim_header meanP \
--pretrained_clip_name ViT-B/32

Reference: Training CLI arguments for main_task_retrieval.py

master

The following arguments are used when running the retrieval training task via main_task_retrieval.py.

#!/bin/bash
# Note: These are descriptions of the flags used in the training commands

--features_path: The video root path
--linear_patch: Set to `2d` or `3d`
--sim_header: Similarity calculation approach. Options: `meanP`, `seqLSTM`, `seqTransf`, or `tightTransf`
--pretrained_clip_name: CLIP model name. Options: `ViT-B/32` or `ViT-B/16`
--resume_model: Reload saved optimizer state to continue training. Requires setting `--init_model` simultaneously.
--do_train: Flag to enable training mode
--num_thread_reader: Number of reader threads
--epochs: Number of training epochs
--batch_size: Training batch size
--n_display: Number of items to display
--train_csv: Path to training CSV file
--val_csv: Path to validation CSV file
--data_path: Path to data JSON file
--output_dir: Directory to save checkpoints
--lr: Learning rate
--max_words: Maximum number of words
--max_frames: Maximum number of frames
--batch_size_val: Validation batch size
--datatype: Dataset type (e.g., `msrvtt`, `msvd`, `lsmdc`, `activity`, `didemo`)
--expand_msrvtt_sentences: Flag for MSRVTT specific sentence expansion
--feature_framerate: Feature framerate
--coef_lr: Learning rate coefficient
--freeze_layer_num: Number of layers to freeze
--slice_framepos: Frame slicing position
--loose_type: Flag for loose type similarity