DOCSEARCHbyAlgolia
Submit Docs

PaddleVideo Documentation

repository·develop·Indexed 23 days ago

https://github.com/paddlepaddle/paddlevideo

A professional-grade video tool library built on PaddlePaddle for the entire video analysis lifecycle, including data annotation, model training, compression, and industrial deployment. It provides implementations and guides for specific applications such as Abnormal Action Detection (using SlowFast+FasterRCNN), Anti-UAV detection (using PP-YOLO), and Basketball Action detection (utilizing a multi-stage pipeline with ppTSM, BMN, and LSTM models).

Tokens
79.1K
Snippets
267
Records
394
Agent score
82%

What's inside PaddleVideo

  1. Overview of VideoTag Large-scale Video Classification Model

    develop

    VideoTag is a large-scale video classification model based on tens of millions of short video data points from Baidu's business. It supports 3,000 practical labels derived from industrial practice and is designed for large-scale short video classification scenarios (millions/billions of videos).

    Core Architecture

    The model uses a two-stage modeling approach:

    1. Image Modeling (Extractor): A large-scale video feature extraction model is trained using a smaller set of video samples (around 100,000).
    2. Sequence Learning (Predictor): A predictor is trained using tens of millions of data points to learn from the feature sequences extracted in the first stage.

    Technical Workflow

    • Data Processing: Videos are decoded into a sequence of image frames, which are then fed into VideoTag for training or prediction.
    • Image Modeling: Uses a TSN (Temporal Segment Networks) architecture to extract features from video frames. Each frame is converted into a feature vector, and a video is represented as a sequence of these vectors.
    • Sequence Learning: Models the feature sequences using techniques like Attention clusters, LSTM, and Nextvlad to learn combinations of features. The provided example code specifically utilizes the Attention_LSTM network for sequence feature prediction.
    • Prediction: Results from multiple models can be fused to improve classification accuracy.

  2. Overview of Ma-Net Video Segmentation Model

    develop

    Ma-Net is a PaddlePaddle implementation of the CVPR 2020 paper "Memory aggregation networks for efficient interactive video object segmentation". It is designed for efficient interactive video object segmentation.

    Currently, the implementation supports:

    • Model training on the DAVIS dataset.
    • Model testing on the DAVIS dataset.
    • Future support for model inference on arbitrary videos.

  3. Overview of PaddleVideo TIPC (Training and Inference Pipeline Certification)

    develop
    TIPC (Training and Inference Pipeline Certification) is a toolset provided for PaddleVideo models to verify the seamless integration of the entire pipeline from training to inference and deployment. It allows users to perform one-click testing to ensure that different training methods (e.g., mixed precision), model compression techniques (e.g., quantization), and deployment modes (e.g., C++, Paddle Serving, Paddle-Lite) are correctly implemented and aligned in terms of prediction accuracy.

  4. Overview of MultimodalVideoTag

    develop

    MultimodalVideoTag is a video classification model designed for multi-label tasks in multimodal scenarios. Developed using PaddlePaddle 2.0, it integrates three modalities: Text (video titles), Video/Image (extracted frame sequences), and Audio (PCM files).

    Key architectural features include:

    • Feature Extraction: Uses pre-trained ResNet for images, VGGish for audio, and ERNIE 1.0 for text.
    • Sequence Learning: Employs independent LSTMs for image and audio sequences, with a TextCNN following ERNIE for text modeling.
    • Multimodal Fusion: Uses text features to guide the LSTM pooling process (assigning temporal weights to image and audio features) via a text_guide mechanism.
    • Prediction: Uses a Sigmoid multi-label classifier to support multiple tags per video.

  5. Introduction to Ma-Net

    develop

    Ma-Net is a PaddlePaddle implementation of the CVPR2020 paper "Memory aggregation networks for efficient interactive video object segmentation". It is designed for efficient interactive video object segmentation tasks.

    Currently, the implementation supports:

    • Model training on the DAVIS dataset.
    • Model testing on the DAVIS dataset.
    • (Upcoming) Model inference on arbitrary videos.

  6. Overview of the FootballAction Model

    develop

    FootballAction is a football action detection algorithm implemented using PaddleVideo. It identifies the start and end times of specific action segments within football match videos and classifies them into 8 categories:

    • Background (背景)
    • Goal (进球)
    • Corner Kick (角球)
    • Free Kick (任意球)
    • Yellow Card (黄牌)
    • Red Card (红牌)
    • Substitution (换人)
    • Throw-in (界外球)

    The algorithm uses a multi-modal approach combining image and audio features through three main stages:

    1. Feature Extraction: Uses PP-TSM for image features and VGGish for audio features.
    2. Proposal Extraction: Uses BMN.
    3. Action Classification & Regression: Uses AttentionLSTM.

  7. Overview of EIVideo

    develop

    EIVideo is an interactive intelligent video annotation toolbox built upon the PaddlePaddle MA-Net interactive video segmentation model. It allows users to complete full-video annotation by labeling only a few frames. If the automatic results are insufficient, users can interact with the video multiple times to continuously improve the segmentation quality.

    For users who prefer a graphical interface over command-line tools, QEIVideo is provided as a GUI wrapper to simplify the annotation process.

  8. Overview of PaddleVideo

    develop

    PaddleVideo is a comprehensive video tool library designed for both academic research and industrial practice in the video domain. It provides a complete pipeline covering data production, model training, compression, and prediction/deployment.

    Key features include:

    • Support for cutting-edge video algorithms.
    • Industrial-grade models like PP-TSM and PP-TSMv2.
    • A full workflow from data annotation to model deployment.
    • Support for various tasks including action recognition, localization, and skeleton-based recognition.

  9. NTU-RGB+D Dataset Overview

    develop

    The NTU-RGB+D dataset is used for skeleton-based action recognition and contains 60 action classes and 56,880 video samples. It supports two primary evaluation splits:

    • Cross-subject (xsub): Divided by character ID. Includes 40,320 training samples and 16,560 test samples.
    • Cross-view (xview): Divided by camera division. Training set includes samples from cameras 2 and 3 (37,930 samples); test set includes samples from camera 1 (18,960 samples).

  10. Explore the PaddleVideo Model Zoo

    develop

    The PaddleVideo Model Zoo provides implementations of various academic algorithms for video analysis tasks. The available models are categorized into several functional groups:

    • Action Recognition: Includes PP series (PP-TSM, PP-TSN, PP-TimeSformer), 2D/3D methods (TSN, TSM, SlowFast, TimeSformer, VideoSwin, TokenShift, AttentionLSTM), and Lite models (MoViNet).
    • Skeleton-based Action Recognition: Uses graph convolutional networks (ST-GCN, AGCN, 2s-AGCN, CTR-GCN).
    • Sequence Action Detection: One-stage methods like BMN.
    • Temporal Segment: Methods for temporal segmentation such as MS-TCN and ASRF.
    • Spatio-temporal Motion Detection: Combined approaches like SlowFast + Fast R-CNN.
    • Multimodal: Models for learning (ActBERT) and retrieval (T2VLAD).
    • Video Target Segmentation: Semi-supervised (CFBI) and supervised (MA-Net) methods.
    • Monocular Depth Estimation: Unsupervised methods like ADDS.

  11. Explore PaddleVideo application cases

    develop

    PaddleVideo provides several specialized application solutions for video analysis and recognition tasks. You can explore the following implementation details for each case in their respective directories:

    • Sports Action Recognition/Detection:
      • FootballAction: Football action detection.
      • BasketballAction: Basketball action detection.
      • TableTennis: Table tennis action recognition.
      • FigureSkating: Figure skating action recognition.
    • Video Classification:
      • VideoTag: Large-scale video classification (3000 categories).
      • MultimodalVideoTag: Multimodal video classification.
    • Specialized Analysis:
      • VideoQualityAssessment: Video quality assessment.
      • PP-Care: 3D MRI medical image recognition.
      • EIVideo: Interactive video segmentation tool.
      • Anti-UAV: UAV (Unmanned Aerial Vehicle) detection.
      • AbnormalActionDetection: Abnormal behavior detection.
      • PP-Human: Pedestrian analysis and action recognition.

  12. Available datasets in PaddleVideo

    develop

    PaddleVideo supports various datasets categorized by computer vision tasks. You can find detailed documentation for each dataset (including download instructions and formatting) in their respective sub-pages. The supported categories include:

    • Action Recognition: Kinetics-400, UCF101, ActivityNet, YouTube-8M
    • Action Localization: ActivityNet
    • Spatio-Temporal Action Detection: AVA
    • Skeleton-based Action Recognition: NTURGB+D, FSD
    • Depth Estimation: Oxford-RobotCar
    • Text-Video Retrieval: MSR-VTT
    • Text-Video Pretrained Model: HowTo100M