DOCSEARCHbyAlgolia
Submit Docs

InternVideo

repository·main·Indexed 25 days ago

https://github.com/opengvlab/internvideo

A repository of video foundation models (InternVideo1 through InternVideo-Next) and large-scale datasets (InternVid) for multimodal video understanding and generation. It includes ViCLIP for video-text representation, VideoChat instruction data, and implementations for downstream tasks such as zero-shot action recognition, video question answering (Video QA), and action detection using BMN and BSN models.

Tokens
103.2K
Snippets
241
Records
442
Agent score
81%

What's inside InternVideo

  1. Overview of InternVideo2

    main

    InternVideo2 is a collection of video foundation models designed for multimodal video understanding. It provides code and models for Stage 1 and Stage 2 of the InternVideo2 training process.

    Key capabilities include:

    • High-performance action recognition (e.g., 92.1% Top1 accuracy on Kinetics 400).
    • SOTA performance across over 60 video and audio-related tasks, including action recognition, temporal localization, and retrieval.

    Note on Stage 3: Training and testing for Stage 3 of InternVideo2 are located in the videochat2 repository.

  2. Overview of InternVideo series

    main

    InternVideo is a collection of video foundation models designed for multimodal understanding. The repository includes several generations of models and datasets:

    • InternVideo (InternVideo1): General video foundation models using generative and discriminative learning.
    • InternVideo2: Scaled video foundation models for multimodal video understanding.
    • InternVideo2.5: Video Multimodal Large Language Models (MLLMs) with long and rich context modeling.
    • InternVideo3: Multimodal contextual reasoning using efficient long-horizon agents.
    • InternVideo-Next: General video foundation models focused on genuine world understanding.
    • InternVid: A large-scale video-text dataset for multimodal understanding and generation.

  3. Overview of InternVideo Foundation Models

    main

    InternVideo is a general video foundation model designed for high-performance video and video-text understanding. It utilizes both generative (masked video modeling) and discriminative (video-language contrastive learning) self-supervised learning to achieve state-of-the-art performance across 39 video datasets.

    Key achievements include:

    • Kinetics-400: 91.1% top-1 accuracy.
    • Something-Something V2: 77.2% top-1 accuracy.
    • Versatility: High performance in action recognition, temporal localization, video retrieval, and open-set applications.

  4. Overview of UniFormerV2

    main
    UniFormerV2 is a video foundation model designed for spatiotemporal learning. It integrates pre-trained Vision Transformers (ViTs) with efficient UniFormer designs, utilizing new local and global relation aggregators to balance accuracy and computational efficiency. It has achieved state-of-the-art performance on several video benchmarks, including Kinetics (400/600/700), Moments in Time, Something-Something (V1/V2), ActivityNet, and HACS.

  5. Overview of InternVideo2.5

    main

    InternVideo2.5 is a video multimodal large language model (MLLM) built upon InternVL2.5. It is specifically enhanced with Long and Rich Context (LRC) modeling to improve the perception of fine-grained details and the capture of long-form temporal structures in videos.

    Key technical features include:

    • Dense vision task annotations using Direct Preference Optimization (TPO).
    • Compact spatiotemporal representations via adaptive hierarchical token compression (HiCo).

    It is designed to outperform existing MLLMs in both short and long video understanding benchmarks, offering capabilities such as object tracking and segmentation.

  6. Overview of VideoMAE features

    main

    This implementation is modified from the original VideoMAE and includes the following enhanced capabilities:

    • Flexible Fine-tuning: Supports adjusting both the input resolution and the number of frames during fine-tuning (the original codebase only supported adjusting the number of frames).
    • Enhanced Pre-training: Supports applying repeated augmentation during the pre-training phase.

  7. Benchmark Environment and Metrics

    main

    The benchmark evaluates training speed (s/iter) for video understanding models. The measured time is the average training time per iteration, including data processing and model training. To avoid device warmup noise, the first 50 iterations are skipped.

    Hardware Environment:

    • 8 NVIDIA Tesla V100 (32G) GPUs
    • Intel(R) Xeon(R) Gold 6146 CPU @ 3.20GHz

    Software Environment:

    • Python 3.7
    • PyTorch 1.4
    • CUDA 10.1
    • CUDNN 7.6.03
    • NCCL 2.4.08

  8. TSN Model Zoo for Action Recognition

    main

    The TSN (Temporal Segment Networks) model zoo provides pre-trained configurations and checkpoints for various video action recognition benchmarks. Users can select models based on the target dataset, backbone architecture (e.g., ResNet50, ResNet101), and input resolution.

    Available datasets include:

    • Kinetics-400
    • Something-Something V1 & V2
    • Moments in Time
    • Multi-Moments in Time
    • ActivityNet v1.3

    Each model entry provides a link to its specific .py configuration file, the pre-trained checkpoint (.pth), training logs, and evaluation JSON results.

  9. Understand the InternVideo project structure

    main

    The InternVideo repository is organized into two main functional areas: Pretrain for foundation model training and Downstream for specific task implementations. Each sub-directory is self-contained, meaning it contains its own dependencies, README, and training/inference scripts.

    Pretraining Components

    • VideoMAE: Video masked autoencoder pretraining and supervised finetuning.
    • Multi-Modalities-Pretraining: Video-language contrastive learning (primarily for demo/inference).
    • ViCLIP: Video CLIP trained on the InternVid-10M dataset.
    • UniFormerV2: Spatiotemporal learning using image ViTs with video UniFormer (provided as a git submodule).

    Downstream Task Implementations

    • Video-Text-Retrieval: Retrieval tasks on benchmarks like MSR-VTT, DiDeMo, LSMDC, MSVD, VATEX, and ActivityNet.
    • Open-Set-Action-Recognition: Recognition on UCF101 and HMDB51.
    • Spatial-Temporal-Action-Localization: Localization on AVA and AVA-Kinetics.
    • Temporal-Action-Localization: Localization on ActivityNet, THUMOS14, HACS, and FineAction.
    • Visual-Language-Navigation: Navigation tasks using VLN-CE.
    • multi-modalities-downstream: Video QA, zero-shot action recognition, and zero-shot multiple choice tasks.
    InternVideo1/
├── Pretrain/                          # Pretraining models
│   ├── VideoMAE/                      # Video masked autoencoder pretraining
│   ├── Multi-Modalities-Pretraining/  # Video-language contrastive learning (demo/inference only)
│   ├── ViCLIP/                        # Video CLIP for transferable video-text representation
│   └── UniFormerV2/                   # Spatiotemporal learning with image ViTs (git submodule)
│
├── Downstream/                        # Downstream task implementations
│   ├── Video-Text-Retrieval/          # Video-text retrieval on 6 benchmarks
│   ├── Open-Set-Action-Recognition/   # Open-set action recognition (MMAction-based)
│   ├── Spatial-Temporal-Action-Localization/  # Spatio-temporal action localization (AVA)
│   ├── Temporal-Action-Localization/  # Temporal action localization (ActivityNet, THUMOS14)
│   ├── Visual-Language-Navigation/    # Vision-language navigation (VLN-CE)
│   └── multi-modalities-downstream/   # VQA, zero-shot action recognition, zero-shot multiple choice
│
└── Media/                             # Images for documentation

  10. What is ViCLIP?

    main

    ViCLIP is a video-text pre-training baseline built upon the CLIP architecture. It consists of a video encoder (ViT) and a text encoder, both initialized from their respective CLIP components.

    Key technical features include:

    • Spatiotemporal Attention: The native attention in the video encoder is updated to spatiotemporal attention.
    • Masked Pre-training: To improve learning efficiency, video masking is applied during pre-training.

  11. Use intermediate variables in configuration files

    main

    The configuration system allows the use of intermediate variables to maintain clean and modular configs. You can define variables for data pipelines (train_pipeline, val_pipeline, test_pipeline), annotation files (ann_file_train, ann_file_val, ann_file_test), data roots (data_root, data_root_val), and image normalization settings (img_norm_cfg). These variables are then passed into the main data dictionary to construct the dataset and augmentation components.

    # Example of using intermediate variables
dataset_type = 'RawframeDataset'
data_root = 'data/kinetics400/rawframes_train'
data_root_val = 'data/kinetics400/rawframes_val'
ann_file_train = 'data/kinetics400/kinetics400_train_list_rawframes.txt'
ann_file_val = 'data/kinetics400/kinetics400_val_list_rawframes.txt'
ann_file_test = 'data/kinetics400/kinetics400_val_list_rawframes.txt'

img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_bgr=False)

train_pipeline = [
    dict(type='SampleFrames', clip_len=32, frame_interval=2, num_clips=1),
    dict(type='RawFrameDecode'),
    dict(type='Resize', scale=(-1, 256)),
    dict(
        type='MultiScaleCrop',
        input_size=224,
        scales=(1, 0.8),
        random_crop=False,
        max_wh_scale_gap=0),
    dict(type='Resize', scale=(224, 224), keep_ratio=False),
    dict(type='Flip', flip_ratio=0.5),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='FormatShape', input_format='NCTHW'),
    dict(type='Collect', keys=['imgs', 'label'], meta_keys=[]),
    dict(type='ToTensor', keys=['imgs', 'label'])
]

# ... val_pipeline and test_pipeline defined similarly ...

data = dict(
    videos_per_gpu=8,
    workers_per_gpu=4,
    train=dict(
        type=dataset_type,
        ann_file=ann_file_train,
        data_prefix=data_root,
        pipeline=train_pipeline),
    val=dict(
        type=dataset_type,
        ann_file=ann_file_val,
        data_prefix=data_root_val,
        pipeline=val_pipeline),
    test=dict(
        type=dataset_type,
        ann_file=ann_file_val,
        data_prefix=data_root_val,
        pipeline=test_pipeline))

  12. Understand the frame calculation formula

    main

    In the UniFormerV2 model zoo, the total number of frames used for inference is calculated using the following formula:

    #Frame = #input_frame x #crop x #clip

    Where:

    • #input_frame: The number of frames input for the model per inference.
    • #crop: The number of spatial crops (e.g., 3 for left/right/center).
    • #clip: The number of temporal clips (e.g., 4 means repeated sampling of four clips with different start indices).