VideoMAE V2 Documentation

Kinetics-710 is a merged dataset created by combining the training and validation sets of Kinetics-400, 600, and 700. Duplicates (based on YouTube IDs) were removed, and validation videos present in the training set were deleted.

Because category names vary across Kinetics versions, labels were grouped using misc/k710_identical_label_merge.json, resulting in 710 unique categories.

Converting Models: You can convert a k710 classification model back to a k400, k600, or k700 model using the provided mapping files in the /misc/ directory (e.g., /misc/label_710to400.json).

To perform multi-node Slurm training, use the provided Slurm scripts (e.g., script/pretrain/vit_g_hybrid_pt.sh). You must configure OUTPUT_DIR and DATA_PATH within the script.

Key Slurm environment variables used in the script:

• GPUS: Total number of GPUs across all nodes (e.g., 64 for 8 nodes x 8 GPUs).
• GPUS_PER_NODE: Number of GPUs per node (e.g., 8).
• PARTITION: The Slurm partition name.
• JOB_NAME: The first positional argument passed to the script.
• PY_ARGS: Any additional arguments passed to run_mae_pretraining.py (passed as positional arguments after the job name).

Example execution:

bash script/pretrain/vit_g_hybrid_pt.sh hybrid_pretrain

Use the extract_tad_feature.py script to extract features from video datasets for use in Temporal Action Detection (TAD) pipelines. This is typically used to replace I3D features with VideoMAE V2 features in baselines like ActionFormer.

To extract features for the THUMOS14 dataset, use the following command structure. Ensure you replace the placeholder paths with your actual local paths for the video data, the destination for extracted features, and the model checkpoint.

python extract_tad_feature.py \
    --data_set THUMOS14 \
    --data_path YOUR_PATH/thumos14_videos \
    --save_path YOUR_PATH/th14_vit_g_16_4 \
    --model vit_giant_patch14_224 \
    --ckpt_path YOUR_PATH/vit_g_hyrbid_pt_1200e_k710_ft.pth

To perform pre-training of the VideoMAE V2 models, follow the procedures outlined in docs/PRETRAIN.md.

The pre-training instruction is in [PRETRAIN.md](docs/PRETRAIN.md).

To install the necessary dependencies and set up the environment, refer to the detailed instructions in docs/INSTALL.md.

Please follow the instructions in [INSTALL.md](docs/INSTALL.md).

For multi-node distributed training without Slurm, use torch.distributed.launch. You must run the script on every node in the cluster.

Key requirements:

• MASTER_PORT: Must be set to the same value on all nodes.
• NODE_RANK: The index of the current node (starting from 0).
• MASTER_ADDR: The IP address of the master node.
• N_NODES: Total number of nodes in the cluster.

Example execution on each node:

# On the first node (rank 0)
NODE_RANK=0 MASTER_ADDR=192.168.1.1 bash dist_train_vit_g_hybrid_pt.sh $NODE_RANK $MASTER_ADDR

# On the second node (rank 1)
NODE_RANK=1 MASTER_ADDR=192.168.1.1 bash dist_train_vit_g_hybrid_pt.sh $NODE_RANK $MASTER_ADDR

# Example command structure for each node
NODE_RANK=0
# MASTER_ADDR should be set as the ip of current node
bash dist_train_vit_g_hybrid_pt.sh $NODE_RANK $MASTER_ADDR

Fine-tuning uses two different dataset implementations depending on the data type. Ensure your data list file matches the required format for the corresponding class:

• VideoClsDataset (for video data): Uses the format video_path label.
• RawFrameClsDataset (for rawframes data): Uses the format frame_folder_path total_frames label.

Use the --data_root ${PATH_PREFIX} flag in training or inference scripts to define the path prefix for the entries in your list.

Note: SSV2 (Something-Something V2) uses RawFrameClsDataset by default.

# VideoClsDataset format (e.g., k400)
your_path/k400/jf7RDuUTrsQ.mp4 325

# RawFrameClsDataset format (e.g., ssv2)
your_path/SomethingV2/frames/74225 62 140

To fine-tune the pre-trained models on specific downstream tasks, follow the instructions provided in docs/FINETUNE.md.

The fine-tuning instruction is in [FINETUNE.md](docs/FINETUNE.md).

For instructions on how to format and prepare your datasets for use with this project, refer to docs/DATASET.md.

Please follow the instructions in [DATASET.md](docs/DATASET.md) for data preparation.

To set up the VideoMAEv2 environment, create a new Conda environment with Python 3.8, install the specific PyTorch versions, and then install the dependencies from requirements.txt.

Important Compatibility Notes:

• PyTorch: It is recommended to use PyTorch >= 1.12.0 to reduce GPU memory usage. While PyTorch 2.0 is supported for pre-training, it has not been fully tested.
• timm: It is highly recommended to install timm == 0.4.12 because several APIs used in this project are deprecated in newer versions of timm.

conda create --name videomae python=3.8 -y
conda activate videomae

conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 -c pytorch

pip install -r requirements.txt