DOCSEARCHbyAlgolia
Submit Docs

ActionCLIP

repository·master·Indexed 20 days ago

https://github.com/sallymmx/actionclip

A PyTorch implementation of ActionCLIP, a video action recognition paradigm leveraging CLIP. It supports training and testing on datasets such as Kinetics-400, UCF101, and HMDB51, featuring ViT-B/32 and ViT-B/16 backbones. The library provides tools for zero-shot validation, various visual prompting strategies (pre-network, in-network, and post-network), and pre-trained model checkpoints.

Tokens
2.6K
Snippets
9
Records
13
Agent score
69%

What's inside ActionCLIP

  1. Use pre-trained models or resume training

    master

    ActionCLIP allows you to load pre-trained weights for the entire network or resume training from a checkpoint.

    • Pre-trained models: Use the pretrain key to provide a path to a pre-trained model file. If pretrain is set to None (default), the system uses the CLIP model as the pre-trained model, provided that config.network.arch is set to ViT-B/32 or ViT-B/16 and config.network.init is set to True. This enables zero-shot validation pre-trained on Kinetics-400.
    • Resuming training: Use the resume key to provide the path to a saved model checkpoint (e.g., a .pth.tar file) to continue training from where it was interrupted.
    # Example pretrain path
pretrain: /path/to/k400_vit32_8frame.pt

# Example resume path
resume: /path/to/checkpoint/3.pth.tar

  2. Install ActionCLIP via Conda

    master

    The easiest way to set up the ActionCLIP environment is by using the provided environment.yml file. This ensures all necessary dependencies, including specific versions of PyTorch and other libraries, are correctly installed.

    Run the following command to create the conda environment:

    conda env create -f environment.yml

  3. Download pretrained ActionCLIP models

    master

    Pretrained models are available for various backbones (e.g., ViT-B/32, ViT-B/16) and frame configurations (8, 16, or 32 frames).

    • For a comprehensive list, see MODEL_ZOO.md.
    • For Kinetics-400 models, specific checkpoints are provided via Baidu Pan links (passwords required).
    • For HMDB51 and UCF101, accuracy is reported using Kinetics-400 pretrained models.

    Kinetics-400 Example Checkpoints:

    modeln-frametop1 Acccheckpointpwd
    ViT-B/32878.36%linkb5ni
    ViT-B/16881.09%linkhqtv

  5. Prerequisites for ActionCLIP

    master

    To use ActionCLIP, ensure your environment meets the following library requirements:

    • PyTorch: version >= 1.8
    • wandb: Weights & Biases
    • RandAugment
    • pprint
    • tqdm
    • dotmap
    • yaml
    • csv

    Additionally, ffmpeg is required for video data pre-processing. For more detailed installation instructions, refer to INSTALL.md.

  6. Configure the training schedule and optimizer

    master

    The solver section controls the learning policy and optimizer settings. When fine-tuning, it is recommended to use a smaller learning rate and fewer epochs.

    Learning Policy

    • type: The decay schedule type (e.g., cosine or multistep).
    • epochs: Total number of training epochs.
    • start_epoch: The epoch to start the policy.

    Optimizer Settings

    • optim: The optimizer algorithm (e.g., adamw, adam, or sgd).
    • lr: The learning rate.
    • lr_warmup_step: Number of steps for learning rate warmup.
    • clip_gradient: Gradient clipping value.
    • loss_type: The loss function used (e.g., nll).
    • weight_decay: Weight decay coefficient.
    • lr_decay_step: Step size for learning rate decay.
    • lr_decay_factor: Factor by which the learning rate is decayed.
    solver:
    # learning policy
    type: cosine       # cosine or multistep
    epochs: 50
    start_epoch: 0
    epoch_offset: 0

    # optimizer 
    optim: adamw      # adam, sgd, or adamw
    clip_gradient: 20
    loss_type: nll
    lr: 5.e-6
    lr_warmup_step: 5
    momentum: 0.9
    weight_decay: 0.0005
    lr_decay_step: 15
    lr_decay_factor: 0.1

  7. Configure dataset settings

    master

    To adapt ActionCLIP to a custom dataset, modify the data section in your configuration. Supported datasets include kinetics400, ucf101, and hmdb51. You must specify the number of classes, the image naming convention, and the file paths for training/validation lists and labels.

    # dataset settings
data:
    dataset: hmdb                                                 # dataset names
    num_classes: 51                                               # dataset classes
    image_tmpl: 'img_{:05d}.jpg'                                  # Picture naming format
    train_list: 'lists/hmdb51/train_rgb_split1.txt'               # dataset training list  
    val_list: 'lists/hmdb51/val_rgb_split1.txt'                   # dataset validation list
    label_list: 'lists/hmdb51_labels.csv'                         # dataset label list

  8. Configure model architecture and visual prompts

    master

    You can define the backbone architecture and the type of visual prompting used in the network.

    Backbone

    Supported architectures (arch) are ViT-B/32 and ViT-B/16.

    Visual Prompts

    ActionCLIP supports several prompting strategies:

    • Pre-network Prompt: Controlled by the tsm flag.
    • In-network Prompt: Controlled by the joint flag.
    • Post-network Prompt: Controlled by the sim_header key. Supported values include seqTransf, meanP, seqLSTM, conv_1D, and seqTransf_cls.
    # model settings
network:
    arch: ViT-B/32  # Backbone: ViT-B/32 or ViT-B/16

# visual prompt settings
network:
    tsm: False                                                  # Pre-network Prompt
    joint: False                                                # In-network Prompt
    sim_header: "seqTransf"                                     # Post-network Prompt (seqTransf, meanP, seqLSTM, conv_1D, seqTransf_cls)

  9. Test pretrained models

    master

    To evaluate downloaded pretrained models on Kinetics, HMDB51, or UCF101, use the scripts/run_test.sh script with the appropriate configuration file.

    Standard Testing

    Run the test script with a standard test config:

    bash scripts/run_test.sh ./configs/k400/k400_test.yaml

    Zero-shot Validation

    ActionCLIP supports zero-shot validation.

    Zero-shot on Kinetics (from CLIP pretrained models):

    bash scripts/run_test.sh ./configs/k400/k400_ft_zero_shot.yaml

    Zero-shot on UCF101 and HMDB51 (from Kinetics pretrained models): Note: You must first prepare the Kinetics-400 pretrained model.

    bash scripts/run_test.sh ./configs/hmdb51/hmdb_ft_zero_shot.yaml
    # test
bash scripts/run_test.sh  ./configs/k400/k400_test.yaml

  10. Train ActionCLIP models

    master

    Training can be performed using the scripts/run_train.sh script. Examples include training from CLIP pretrained models or fine-tuning from Kinetics-400 pretrained models.

    Training Examples

    Train on Kinetics (from CLIP pretrained models):

    bash scripts/run_train.sh ./configs/k400/k400_train.yaml

    Train on HMDB51 (from Kinetics400 pretrained models):

    bash scripts/run_train.sh ./configs/hmdb51/hmdb_train.yaml

    Train on UCF101 (from Kinetics400 pretrained models):

    bash scripts/run_train.sh ./configs/ucf101/ucf_train.yaml

    For detailed training configurations and support for custom datasets, refer to configs/README.md.

    # train 
bash scripts/run_train.sh  ./configs/k400/k400_train.yaml

  11. Download pre-trained ActionCLIP models for Kinetics-400

    master

    ActionCLIP provides several pre-trained models for the Kinetics-400 dataset using different visual backbones and frame configurations. The recommended visual prompt is the Transformer (Trans) backbone.

    Available models include:

    • ViT-B/32 with 8 frames
    • ViT-B/16 with 8 frames
    • ViT-B/16 with 16 frames
    • ViT-B/16 with 32 frames
    | model | n-frame | top1 Acc(single-crop) | top5 Acc(single-crop) |
| :---: | :---: | :---: | :---: |
| ViT-B/32 | 8 | 78.36% | 94.25% |
| ViT-B/16 | 8 | 81.09% | 95.49% |
| ViT-B/16 | 16 | 81.68% | 95.87% |
| ViT-B/16 | 32 | 82.32% | 96.20% |

  12. Download pre-trained ActionCLIP models for UCF101 and HMDB51

    master

    For the UCF101 and HMDB51 datasets, ActionCLIP provides models pre-trained on Kinetics-400. These are reported under the 'accurate' setting.

    UCF101

    • ViT-B/16 with 32 frames (Top-1 Accuracy: 97.1%)

    HMDB51

    • ViT-B/16 with 32 frames (Top-1 Accuracy: 76.2%)
    ### UCF101
| model | n-frame | top1 Acc(single-crop) |
| :---: | :---: | :---: |
| ViT-B/16 | 32 | 97.1% |

### HMDB51
| model | n-frame | top1 Acc(single-crop) |
| :---: | :---: | :---: |
| ViT-B/16 | 32 | 76.2% |