DOCSEARCHbyAlgolia
Submit Docs

ActionFormer

repository·main·Indexed 20 days ago

https://github.com/happyharrycn/actionformer_release

A Transformer-based model for temporal action localization designed to detect onsets, offsets, and categories of action instances in untrimmed videos. The repository includes implementations for datasets such as THUMOS14, ActivityNet 1.3, EPIC Kitchens 100, and the Ego4D Moment Queries Benchmark, supporting various feature types including I3D, TSP, SlowFast, Omnivore, and EgoVLP.

Tokens
2.8K
Snippets
10
Records
13
Agent score
19%

What's inside ActionFormer

  1. ActionFormer Code Overview

    main

    The ActionFormer repository follows a structure inspired by Detectron2. The core logic is organized into the following modules:

    • ./libs/core: Handles parameter configuration.
    • ./libs/datasets: Contains the data loader and I/O modules.
    • ./libs/modeling: Contains the main model architecture and its building blocks.
    • ./libs/utils: Provides utility functions for training, inference, and postprocessing.

  3. Reproduce results on Ego4D Moment Queries Benchmark

    main

    To reproduce results on the Ego4D benchmark, follow these steps:

    1. Download Features and Annotations: Obtain official SlowFast and Omnivore features from the Ego4D website and official EgoVLP features from this link.
    2. Convert Data: Use the provided script ./tools/convert_ego4d_trainval.py to convert the downloaded features and annotations.
    3. Organize Directory Structure: Unpack the converted data into a ./data/ego4d/ directory with the following subfolders:
      • `annotations/
      • `slowfast_features/
      • `omnivore_features/
      • `egovlp_features/
    4. Train the Model: Use a configuration file (e.g., ego4d_omnivore_egovlp.yaml) to start training. Training results, logs, and checkpoints will be stored in the ./ckpt directory.
    5. Evaluate: Run the evaluation script using the training config and the checkpoint directory.

    Hardware Requirements:

    • Training on Ego4D with all three features requires ~4.5GB GPU memory.
    • Inference may require over 10GB GPU memory. A GPU with at least 12 GB of memory is recommended.
    # Train the model
python ./train.py ./configs/ego4d_omnivore_egovlp.yaml --output reproduce

# Evaluate the trained model
python ./eval.py ./configs/ego4d_omnivore_egovlp.yaml ./ckpt/ego4d_omnivore_egovlp_reproduce

  4. Evaluate Pre-trained ActionFormer model on THUMOS14

    main

    You can evaluate a pre-trained model for THUMOS14 without training it yourself.

    1. Setup Pre-trained Files

    1. Create a ./pretrained folder.
    2. Download the pre-trained model and logs from the provided Google Drive link.
    3. Unpack the files into ./pretrained. The structure must be:
    project_root/
├── pretrained/
│   └── thumos_i3d_reproduce/
│       ├── thumos_reproduce_log.txt
│       ├── thumos_reproduce_results.txt
│       ├── config.txt
│       └── epoch_034.pth.tar
└── libs/

    2. Run Evaluation

    Use the eval.py script pointing to the pre-trained directory:

    python ./eval.py ./configs/thumos_i3d.yaml ./pretrained/thumos_i3d_reproduce/

    Expected results (mAP at tIoUs):

    Method0.30.40.50.60.7Avg
    ActionFormer82.1377.8070.9559.4043.8766.83

  5. Reproduce results on EPIC Kitchens 100

    main

    ActionFormer uses separate models for nouns and verbs on the EPIC Kitchens 100 dataset.

    1. Download Data: Download epic_kitchens.tar.gz (MD5: add9803756afd9a023bc9a9c547e0229) and unpack into ./data/epic_kitchens/.
    2. Training: Use the specific config for verbs or nouns.
    3. Evaluation: Use the corresponding config and checkpoint path.

    Hardware Requirements: Training requires ~4.5GB GPU memory; inference may require >10GB. A GPU with at least 12GB is recommended.

    # --- VERB MODEL ---
# Train
python ./train.py ./configs/epic_slowfast_verb.yaml --output reproduce
# Evaluate
python ./eval.py ./configs/epic_slowfast_verb.yaml ./ckpt/epic_slowfast_verb_reproduce

# --- NOUN MODEL ---
# Train
python ./train.py ./configs/epic_slowfast_noun.yaml --output reproduce
# Evaluate
python ./eval.py ./configs/epic_slowfast_noun.yaml ./ckpt/epic_slowfast_noun_reproduce

  6. Evaluate pre-trained Ego4D models

    main

    To evaluate the provided pre-trained models:

    1. Download Models: Download pre-trained models from the Google Drive link.
    2. Setup Directory: Create a ./pretrained folder and unpack the models. The structure should follow this pattern:
      pretrained/
  └───[experiment_name]/
n         ├───config.txt
      ├───ego4d_..._log.txt
      ├───logs/
      └───epoch_010.pth.tar
    3. Run Evaluation: Execute eval.py passing the original configuration file and the path to the specific pre-trained experiment folder.
    python ./eval.py ./configs/ego4d_omnivore_egovlp.yaml ./pretrained/ego4d_omnivore_egovlp_reproduce/

  8. Evaluate pre-trained ActivityNet 1.3 models (TSP or I3D)

    main

    You can evaluate pre-trained models for ActivityNet 1.3 without training them yourself.

    TSP Features

    1. Download the pre-trained model from Google Drive.
    2. Unpack into a ./pretrained/anet_tsp_reproduce/ folder.
    3. Run evaluation:
    python ./eval.py ./configs/anet_tsp.yaml ./pretrained/anet_tsp_reproduce/

    I3D Features

    1. Download anet_1.3_i3d.tar.gz (MD5: e649425954e0123401650312dd0d56a7) and unpack into ./data/anet_1.3/.
    2. To train from scratch: python ./train.py ./configs/anet_i3d.yaml --output reproduce.
    3. To evaluate a pre-trained I3D model: unpack the downloaded model into ./pretrained/anet_i3d_reproduce/ and run:
    python ./eval.py ./configs/anet_i3d.yaml ./pretrained/anet_i3d_reproduce/
    # Evaluate pre-trained TSP model
python ./eval.py ./configs/anet_tsp.yaml ./pretrained/anet_tsp_reproduce/

# Evaluate pre-trained I3D model
python ./eval.py ./configs/anet_i3d.yaml ./pretrained/anet_i3d_reproduce/

  9. Compile the NMS C++ extensions

    main

    The Non-Maximum Suppression (NMS) component is implemented in C++. You must compile these utilities using the provided setup script.

    Note: You must recompile this code every time you update your PyTorch version.

    cd ./libs/utils
python setup.py install --user
cd ../..

  10. Reproduce results on ActivityNet 1.3 using TSP features

    main

    To reproduce results on ActivityNet 1.3 using TSP features, follow these steps:

    1. Download Data: Download anet_1.3.tar.gz (MD5: c415f50120b9425ee1ede9ac3ce11203) from the provided Box, Google Drive, or BaiduYun links. This contains TSP features and action annotations.
    2. Unpack Data: Unpack the contents into a ./data/anet_1.3/ directory. The structure must include ./data/anet_1.3/annotations and ./data/anet_1.3/tsp_features.
    3. Train: Run the training script using the anet_tsp.yaml config.
    4. Monitor: (Optional) Use TensorBoard to monitor training logs.
    5. Evaluate: Run the evaluation script using the trained checkpoint.

    Hardware Requirements: Training requires ~4.6GB GPU memory; inference may require >10GB. A GPU with at least 12GB is recommended.

    # Train
python ./train.py ./configs/anet_tsp.yaml --output reproduce

# Monitor (Optional)
tensorboard --logdir=./ckpt/anet_tsp_reproduce/logs

# Evaluate
python ./eval.py ./configs/anet_tsp.yaml ./ckpt/anet_tsp_reproduce

  11. Evaluate pre-trained EPIC Kitchens 100 models

    main

    To evaluate pre-trained models for EPIC Kitchens 100:

    1. Download the verb or noun model from the provided Google Drive links.
    2. Unpack into a ./pretrained/ subdirectory (e.g., ./pretrained/epic_slowfast_verb_reproduce/).
    3. Run the evaluation command using the appropriate config.

    Folder Structure Example:

    pretrained/
└───epic_slowfast_verb_reproduce/
    ├───epic_slowfast_verb_reproduce_log.txt
    ├───epic_slowfast_verb_reproduce_results.txt
    └───epoch_020.pth.tar
    # Evaluate pre-trained Verb model
python ./eval.py ./configs/epic_slowfast_verb.yaml ./pretrained/epic_slowfast_verb_reproduce/

# Evaluate pre-trained Noun model
python ./eval.py ./configs/epic_slowfast_noun.yaml ./pretrained/epic_slowfast_noun_reproduce/

  12. Reproduce ActionFormer results on THUMOS14

    main

    To reproduce the results on the THUMOS14 dataset, follow these steps:

    1. Download Features and Annotations

    Download thumos.tar.gz (MD5: 375f76ffbf7447af1035e694971ec9b2) from the provided Box, Google Drive, or BaiduYun links. This file contains I3D features, JSON action annotations, and external classification scores.

    2. Unpack Data

    Unpack the downloaded file into a ./data directory. The required structure is:

    project_root/
├── data/
│   └── thumos/
│       ├── annotations/
│       └── i3d_features/
└── libs/

    3. Training

    Run the training script using the I3D configuration. This will generate an experiment folder under ./ckpt containing configs, logs, and checkpoints.

    python ./train.py ./configs/thumos_i3d.yaml --output reproduce

    Note: Training requires ~4.5GB GPU memory. It is recommended to use a GPU with at least 12GB of memory.

    4. Evaluation

    Evaluate the trained model. Expected average mAP should be >66.0%.

    python ./eval.py ./configs/thumos_i3d.yaml ./ckpt/thumos_i3d_reproduce

    Note: Inference may require over 10GB of GPU memory.

    # Train
python ./train.py ./configs/thumos_i3d.yaml --output reproduce

# Evaluate
python ./eval.py ./configs/thumos_i3d.yaml ./ckpt/thumos_i3d_reproduce