PySlowFast

PySlowFast provides support for MViTv2 specifically for video action recognition and detection tasks. For other vision tasks, use the following specialized repositories:

• Image Classification: Use the MViTv2 for image classification repository.
• Object Detection and Instance Segmentation: Use the MViTv2 in Detectron2 project.

PySlowFast provides wrappers for PyTorchVideo models and datasets, allowing you to use the standard PySlowFast workflow (training and testing) with PyTorchVideo components via the PySlowFast config system.

Supported PyTorchVideo Models:

• I3D
• C2D
• R(2+1)D
• CSN
• Slow, SlowFast
• X3D

Supported PyTorchVideo Datasets:

• Kinetics
• Charades
• Something-something v2

To use X3D models, you can find configuration files in configs/Kinetics or refer to MODEL_ZOO.md for pre-trained models.

To train and test an extra small (XS) X3D model on your own dataset, use the tools/run_net.py script with the appropriate config and dataset path.

python tools/run_net.py \
  --cfg configs/Kinetics/X3D-XS.yaml \
  DATA.PATH_TO_DATA_DIR path_to_your_dataset \

To perform self-supervised pre-training (PT) for MaskFeat ViT models on the ImageNet-1K dataset, use the tools/run_net.py script with the appropriate configuration from configs/masked_ssl/.

To fine-tune (FT) a pre-trained model, provide the path to the pre-trained checkpoint using the TRAIN.CHECKPOINT_FILE_PATH configuration key.

# Pre-train ViT-B on ImageNet
python tools/run_net.py \
  --cfg configs/masked_ssl/in1k_VIT_B_MaskFeat_PT.yaml \
  DATA.PATH_TO_DATA_DIR path_to_your_ImageNet_dataset

# Fine-tune ViT-B on ImageNet
python tools/run_net.py \
  --cfg configs/masked_ssl/in1k_VIT_B_FT.yaml \
  DATA.PATH_TO_DATA_DIR path_to_your_ImageNet_dataset \
  TRAIN.CHECKPOINT_FILE_PATH path_to_your_pretrain_checkpoint

PySlowFast provides visualization tools for several stages of the machine learning workflow, including:

• Training, evaluation, and testing processes
• Model analysis
• Running inference with trained models

For detailed usage instructions, see VISUALIZATION_TOOLS.md.

To perform self-supervised pre-training (PT) for MaskFeat MViT models on the Kinetics-400 dataset, use the tools/run_net.py script with the appropriate configuration from configs/masked_ssl/.

To fine-tune (FT) a pre-trained model, provide the path to the pre-trained checkpoint using the TRAIN.CHECKPOINT_FILE_PATH configuration key.

# Pre-train MViT-L on Kinetics-400
python tools/run_net.py \
  --cfg configs/masked_ssl/k400_MVITv2_L_16x4_MaskFeat_PT.yaml \
  DATA.PATH_TO_DATA_DIR path_to_your_Kinetics_dataset

# Fine-tune MViT-L on Kinetics-400
python tools/run_net.py \
  --cfg configs/masked_ssl/k400_MVITv2_L_16x4_FT.yaml \
  DATA.PATH_TO_DATA_DIR path_to_your_Kinetics_dataset \
  TRAIN.CHECKPOINT_FILE_PATH path_to_your_pretrain_checkpoint

To fine-tune a pre-trained MAE model, use the tools/run_net.py script with a fine-tuning configuration file (e.g., from configs/masked_ssl/). You must provide the path to your Kinetics dataset via DATA.PATH_TO_DATA_DIR and the path to your pre-trained checkpoint via TRAIN.CHECKPOINT_FILE_PATH.

python tools/run_net.py \
  --cfg configs/masked_ssl/k400_VIT_L_16x4_FT.yaml \
  DATA.PATH_TO_DATA_DIR path_to_your_Kinetics_dataset \
  TRAIN.CHECKPOINT_FILE_PATH path_to_your_pretrain_checkpoint

To use the Charades dataset, follow these steps:

1. Download the Charades RGB frames from the dataset provider.
2. Download the frame lists for train.csv and val.csv from the provided links.
3. Configure your data settings:
   • Set DATA.PATH_TO_DATA_DIR to the folder containing the frame lists.
   • Set DATA.PATH_PREFIX to the folder containing the RGB frames.

To install PySlowFast, clone the repository and build it using setup.py. You must also add the repository to your $PYTHONPATH to ensure the modules are discoverable.

1. Clone the repository.
2. Build and install in development mode.
3. Update $PYTHONPATH.

git clone https://github.com/facebookresearch/slowfast
cd SlowFast
python setup.py build develop

# Add to PYTHONPATH
export PYTHONPATH=/path/to/SlowFast/slowfast:$PYTHONPATH

To perform unsupervised spatiotemporal representation learning using techniques like MoCo, BYOL, SimCLR, or SwAV, use the configuration files located in configs/contrastive_ssl.

To train a model (e.g., a MoCo R50 Slow-only model with 8x8 sampling) on your own dataset, run the tools/run_net.py script with the appropriate configuration file and dataset path.

python tools/run_net.py \
  --cfg configs/Kinetics/contrastive_ssl/MoCo_SlowR50_8x8.yaml \
  DATA.PATH_TO_DATA_DIR path_to_your_dataset \