DOCSEARCHbyAlgolia
Submit Docs

PyTorchVideo

repository·main·Indexed 25 days ago

https://github.com/facebookresearch/pytorchvideo

A deep learning library for video understanding research built on PyTorch. It provides modular and efficient components including a model zoo of pretrained video models, extensive video datasets, and video-specific transforms. The project also includes PyTorchVideo Trainer, a PyTorch-Lightning based trainer for action recognition, self-supervised learning, and planned action detection.

Tokens
22.8K
Snippets
68
Records
106
Agent score
86%

What's inside PyTorchVideo

  1. Overview of PyTorchVideo/Accelerator

    main

    PyTorchVideo/Accelerator (Accelerator) is a framework designed to accelerate video understanding models across various hardware tiers, from mobile phones to GPUs. It facilitates the design, fine-tuning, optimization, and deployment of hardware-aware efficient video understanding models.

    Key capabilities include:

    • Efficient Model Design: Use carefully tuned efficient blocks for target hardware.
    • Fine-tuning: Fine-tune efficient models from the provided Model Zoo.
    • Optimization: Optimize model kernels and graphs for specific target devices.
    • Deployment: Deploy efficient models to target hardware.

    Accelerator supports int8 operations, which enables significant latency reductions compared to vanilla PyTorch fp32 implementations on mobile devices.

  2. Overview of PyTorchVideo

    main

    PyTorchVideo is a deep learning library designed for video understanding research. It is built on top of PyTorch and provides modular, efficient components including:

    • Video Models: A reproducible model zoo containing state-of-the-art pretrained video models and benchmarks.
    • Video Datasets: Extensive data loaders supporting various datasets.
    • Video Transforms: Video-specific transforms and efficient components designed for accelerated inference on hardware.

  3. Use pytorchvideo.layers modules

    main

    The pytorchvideo.layers package provides specialized neural network layers designed for video processing tasks. You can import specific layer modules to build video models. Available modules include:

    • batch_norm: Batch normalization layers.
    • convolutions: Specialized convolution layers for video data.
    • fusion: Layers for fusing different feature representations.
    • mlp: Multi-Layer Perceptron layers.
    • nonlocal_net: Non-local network layers for capturing long-range dependencies.
    • positional_encoding: Layers for adding positional information to features.
    • swish: Swish activation layers.
    • squeeze_excitation: Squeeze-and-excitation blocks for channel-wise attention.

  6. Load a pre-trained PyTorchVideo model via Torch Hub

    main

    You can load pre-trained video classification models (trained on Kinetics 400) using torch.hub.load. Specify the repository facebookresearch/pytorchvideo:main, the desired model name, and set pretrained=True. After loading, move the model to your target device (e.g., cpu or cuda) and set it to evaluation mode using .eval().

    # Device on which to run the model
device = "cpu"

# Pick a pretrained model and load the pretrained weights
model_name = "slowfast_r50"
model = torch.hub.load("facebookresearch/pytorchvideo:main", model=model_name, pretrained=True)

# Set to eval mode and move to desired device
model = model.to(device)
model = model.eval()

  7. Deploy a quantized (int8) model to mobile

    main

    Efficient blocks in PyTorchVideo are designed to be quantization-friendly. Fusion is automatically handled during convert_to_deployable_form. To deploy an int8 model, follow these steps:

    1. Wrap the model: Use torch.quantization.QuantStub and DeQuantStub to wrap the deployable model.
    2. Configure: Set the qconfig (e.g., using qnnpack for mobile) and set the quantization engine.
    3. Prepare: Use torch.quantization.prepare.
    4. Calibrate: Feed a calibration dataset through the prepared model (step skipped in example).
    5. Convert: Use torch.quantization.convert to produce the quantized model.
    6. Export: Trace the quantized model and apply optimize_for_mobile.
    import torch
import torch.nn as nn
from torch.utils.mobile_optimizer import optimize_for_mobile

# 1. Wrapper for Quantization
class quant_stub_wrapper(nn.Module):
    def __init__(self, module_in):
        super().__init__()
        self.quant = torch.quantization.QuantStub()
        self.model = module_in
        self.dequant = torch.quantization.DeQuantStub()
    def forward(self, x):
        x = self.quant(x)
        x = self.model(x)
        x = self.dequant(x)
        return x

# 2. Setup and Preparation
net_inst_quant_stub_wrapper = quant_stub_wrapper(net_inst_deploy)
net_inst_quant_stub_wrapper.qconfig = torch.quantization.torch.quantization.get_default_qconfig("qnnpack")
torch.backends.quantized.engine = "qnnpack"
net_inst_quant_stub_wrapper_prepared = torch.quantization.prepare(net_inst_quant_stub_wrapper)

# 3. Calibration (Manual step required)
# ... feed calibration data ...

# 4. Conversion
net_inst_quant_stub_wrapper_quantized = torch.quantization.convert(net_inst_quant_stub_wrapper_prepared)

# 5. Export and Optimize
traced_model_int8 = torch.jit.trace(net_inst_quant_stub_wrapper_quantized, input_tensor, strict=False)
traced_model_int8_opt = optimize_for_mobile(traced_model_int8)
# traced_model_int8_opt.save("model_int8.pt")

  9. Use PyTorchVideo models with PySlowFast or PyTorch Lightning

    main

    PyTorchVideo models and datasets are compatible with other major frameworks:

    • PySlowFast: Use the PySlowFast workflow to train or test PyTorchVideo models/datasets.
    • PyTorch Lightning: You can build training and testing pipelines for PyTorchVideo models and datasets using PyTorch Lightning.

  10. Install PyTorchVideo from a local clone

    main

    To install from a local clone (useful for development), clone the repository, navigate to the directory, and use pip install -e .. For a full development and testing environment, include the [test,dev] extras.

    git clone https://github.com/facebookresearch/pytorchvideo.git
cd pytorchvideo

# Standard editable install
pip install -e .

# Install with test and dev extras
pip install -e . [test,dev]

  12. Setup AVA label mapping and VideoVisualizer

    main

    To map predicted class IDs to human-readable action labels, download the AVA V2.2 action list and use AvaLabeledVideoFramePaths to create a mapping. You can then use VideoVisualizer to draw bounding boxes and labels on your video frames.

    Steps:

    1. Download ava_action_list.pbtxt.
    2. Use AvaLabeledVideoFramePaths.read_label_map to parse it.
    3. Initialize VideoVisualizer with the number of classes, the label map, and desired threshold/mode.
    # Download the action text to id mapping
!wget https://dl.fbaipublicfiles.com/pytorchvideo/data/class_names/ava_action_list.pbtxt

# Create an id to label name mapping
label_map, allowed_class_ids = AvaLabeledVideoFramePaths.read_label_map('ava_action_list.pbtxt')

# Create a video visualizer
video_visualizer = VideoVisualizer(81, label_map, top_k=3, mode="thres", thres=0.5)