DOCSEARCHbyAlgolia
Submit Docs

MMAction2 Documentation

repository·main·Indexed 26 days ago

https://github.com/open-mmlab/mmaction2

An open-source video understanding toolbox built on PyTorch and part of the OpenMMLab project. It provides modular implementations for action recognition, localization, and video retrieval, featuring models such as ACRN, LFB, SlowFast, and SlowOnly for datasets including AVA2.1, AVA2.2, and MultiSports.

Tokens
138.7K
Snippets
365
Records
628
Agent score
89%

What's inside MMAction2

  1. Overview of Temporal Shift Module (TSM)

    main

    TSM (Temporal Shift Module) is a method for efficient video understanding that enables 2D CNNs to capture temporal relationships without the high computational cost of 3D CNNs. It works by shifting a portion of the channels along the temporal dimension, facilitating information exchange between neighboring frames.

    Key characteristics:

    • Efficiency: Achieves performance comparable to 3D CNNs while maintaining the complexity of 2D CNNs.
    • Zero Overhead: Can be inserted into 2D CNNs with zero additional computation and zero additional parameters.
    • Online Capability: Supports online settings for real-time, low-latency video recognition and object detection.

  2. Overview of MMAction2

    main

    MMAction2 is an open-source PyTorch-based toolbox for video understanding. It supports multiple major directions including:

    • Action Recognition
    • Skeleton-based Action Recognition
    • Spatio-temporal Action Detection
    • Temporal Action Localization

    Key features include:

    • Full-pipeline support: Implements state-of-the-art models for various video understanding tasks.
    • Modular design: Allows users to define and reuse specific modules within models.
    • Extensive utility tools: Provides visualizers, validation scripts, and evaluators for troubleshooting, fine-tuning, and model comparison.
    • OpenMMLab ecosystem integration: Follows strict development standards and interface conventions, enabling easy switching between OpenMMLab libraries and cross-domain research.

  3. Overview of MMAction2 features

    main

    MMAction2 is an open-source PyTorch-based toolbox for video understanding. Key features include:

    • Modular design: Components are decomposed so users can easily construct customized video understanding frameworks.
    • Five major video understanding tasks:
      • Action recognition
      • Action localization
      • Spatio-temporal action detection
      • Skeleton-based action detection
      • Video retrieval
    • Well tested and documented: Includes detailed documentation, API references, and unit tests.

  4. Overview of MMAction2 Dataset Hierarchy

    main

    MMAction2 uses a hierarchical dataset structure where task-specific classes inherit from base classes. To create a new task-specific dataset, you primarily need to implement the load_data_list(self) method to generate a list of data samples from your annotation files. The base classes handle the rest of the data loading and transformation logic.

    Inheritance Hierarchy

    • MMAction2::VideoDataset / RawframeDataset: Used for action recognition. Requires implementing load_data_list(self).
    • MMAction2::AVADataset: Used for spatiotemporal action detection. Requires implementing load_data_list(self) and often overrides get_data_info(self, idx) to format detection-specific fields.
    • MMAction2::PoseDataset: Used for skeleton-based action recognition.
    • MMAction2::BaseActionDataset: Provides get_data_info(self, idx) to retrieve a sample from the data list.
    • MMEngine::BaseDataset: The root class that implements __getitem__(self, idx), which orchestrates calling get_data_info and applying the pipeline (transformations/augmentations).

  5. Overview of the Gesture Recognition Pipeline

    main

    The Gesture Recognition project implements a lightweight, three-stage skeleton-based pipeline designed for real-time performance on CPU devices. The pipeline follows these stages:

    1. Hand Detection: Outputs bounding boxes of human hands from video frames.
    2. Pose Estimation: Generates keypoints for the detected hands.
    3. Gesture Recognition: Classifies hand actions based on the provided hand skeleton.

    For details on training data preparation and training scripts, refer to TRAINING.md in the project directory.

  6. Overview of ST-GCN for Skeleton-Based Action Recognition

    main
    ST-GCN (Spatial-Temporal Graph Convolutional Networks) is a model designed for skeleton-based action recognition. Unlike conventional methods that rely on hand-crafted parts or traversal rules, ST-GCN automatically learns both spatial and temporal patterns from skeleton data, providing higher expressive power and better generalization. It has been evaluated on large datasets such as Kinetics and NTU-RGBD.

  7. Overview of UniFormerV2

    main
    UniFormerV2 is a family of video networks designed for spatiotemporal representation learning. It improves upon the original UniFormer by arming pretrained Vision Transformers (ViTs) with efficient local and global relation aggregators. This approach allows the model to leverage existing, well-pretrained image ViTs while effectively tackling local video redundancy through a unified convolution and self-attention mechanism. UniFormerV2 achieves state-of-the-art performance across various video benchmarks, including Kinetics-400/600/700, Moments in Time, Something-Something V1/V2, ActivityNet, and HACS.

  9. Overview of MMAction2 capabilities

    main

    MMAction2 is a PyTorch-based open-source toolkit for video understanding. It supports several key tasks:

    • Action Recognition
    • Skeleton-based Action Recognition
    • Spatio-temporal Action Detection
    • Temporal Action Localization

    Key features include a modular design for defining and reusing model components, a variety of analysis tools (visualizers, validation scripts, evaluators), and seamless integration with other OpenMMLab projects (like MMClassification) due to unified interfaces.

  10. Overview of SlowOnly models

    main

    SlowOnly is an implementation of SlowFast networks for video recognition. The architecture consists of two pathways:

    • Slow pathway: Operates at a low frame rate to capture spatial semantics.
    • Fast pathway: Operates at a high frame rate to capture motion at fine temporal resolution. The Fast pathway is designed to be lightweight by reducing channel capacity while still learning useful temporal information.

    These models are used for both action classification and detection in video benchmarks such as Kinetics, Charades, and AVA.

  11. Explore supported models in MMAction2

    main

    MMAction2 supports a wide range of models across different video analysis tasks. You can find detailed configurations and results for these models in the Model Zoo. Supported tasks include:

    • Action Recognition: Includes architectures like C3D, TSN, I3D, SlowFast, VideoMAE, and VideoMAE V2.
    • Temporal Action Localization: Includes BSN, BMN, and TCANet.
    • Spatio-temporal Action Detection: Includes ACRN, SlowOnly+Fast R-CNN, and VideoMAE.
    • Skeleton-based Action Recognition: Includes ST-GCN, 2s-AGCN, PoseC3D, and CTRGCN.
    • Video Retrieval: Includes CLIP4Clip.

  12. Design and structure of the Data Pipeline

    main

    The data pipeline in MMAction2 is a sequence of data transforms used to process a data sample dictionary when indexing from a dataset. Each transform accepts a dict as input, processes it, and returns a dict for the next transform in the sequence.

    Common pipeline steps include:

    1. Loading: Initializing video reading (e.g., DecordInit) or decoding frames.
    2. Sampling: Selecting specific frames (e.g., SampleFrames).
    3. Augmentation: Applying spatial transforms like Resize, RandomResizedCrop, or Flip.
    4. Formatting: Shaping data (e.g., FormatShape) and packing inputs for the model (e.g., PackActionInputs).

    A full list of available transforms is located in mmaction.datasets.transforms.

    train_pipeline = [
    dict(type='DecordInit',),
    dict(type='SampleFrames', clip_len=32, frame_interval=2, num_clips=1),
    dict(type='DecordDecode'),
    dict(type='Resize', scale=(-1, 256)),
    dict(type='RandomResizedCrop'),
    dict(type='Resize', scale=(224, 224), keep_ratio=False),
    dict(type='Flip', flip_ratio=0.5),
    dict(type='FormatShape', input_format='NCTHW'),
    dict(type='PackActionInputs')
]