DOCSEARCHbyAlgolia
Submit Docs

mmtracking

repository·master·Indexed 26 days ago

https://github.com/open-mmlab/mmtracking

A toolbox for multi-object tracking (MOT) providing state-of-the-art algorithms and model configurations. It includes implementations of ByteTrack, SORT, DeepSORT, OC-SORT, QDTrack, and Tracktor, with pre-trained models and configs for datasets such as MOT15, MOT16, MOT17, MOT20, LVIS, and TAO.

Tokens
51.1K
Snippets
105
Records
215
Agent score
87%

What's inside mmtracking

  1. Overview of ByteTrack Multi-Object Tracking

    master
    ByteTrack is a Multi-Object Tracking (MOT) method that improves tracking performance by associating every detection box, including those with low detection scores (e.g., occluded objects), rather than only high-score ones. It uses the similarity between low-score detections and existing tracklets to recover true objects while filtering out background noise. This approach helps reduce true object missing and fragmented trajectories.

  2. Overview of OC-SORT (Observation-Centric SORT)

    master
    OC-SORT is a Multi-Object Tracking (MOT) method designed to be robust against occlusions and non-linear motion. Unlike traditional motion models that assume linear motion and require continuous observations, OC-SORT emphasizes the role of 'observations' to recover tracks after they have been lost, reducing errors accumulated by linear motion models during occlusion periods. It is an online, real-time method that achieves state-of-the-art performance on datasets like MOT17, MOT20, KITTI Pedestrian Tracking, and DanceTrack.

  3. Overview of SELSA for Video Object Detection

    master
    SELSA (Sequence Level Semantics Aggregation) is a method designed for Video Object Detection (VID) that aggregates features across the full sequence to create more robust and discriminative features. Unlike methods relying on optical flow or RNNs, SELSA avoids complex post-processing like Seq-NMS or Tubelet rescoring, maintaining a simpler pipeline. It is particularly effective at addressing appearance degradation caused by fast motion in video frames.

  4. Deep Feature Flow (DFF) for Video Recognition

    master
    Deep Feature Flow (DFF) is a fast and accurate framework for video recognition tasks such as object detection and semantic segmentation. It optimizes performance by running expensive convolutional sub-networks only on sparse key frames and propagating deep feature maps to other frames using a flow field. This approach provides significant speedup compared to per-frame evaluation.

  5. Use Temporal RoI Align for Video Object Recognition

    master

    Temporal RoI Align is a method designed to improve video object detection and instance segmentation by aggregating temporal information from multiple frames. Unlike standard ROI Align which extracts features from a single-frame feature map, Temporal RoI Align utilizes feature similarity to extract features from other frames in the video for current frame proposals, helping mitigate issues caused by appearance deterioration in specific frames.

    Pre-trained models and configurations are available for the ImageNet VID dataset using different backbones (ResNet-50, ResNet-101, and ResNet-X-101).

  6. Understand Single-Image vs. Multi-Image Data Pipelines

    master

    MMTracking supports two types of data pipelines:

    1. Single-image pipeline: Similar to MMDetection, used for standard image-based tasks.
    2. Multi-image (Sequence) pipeline: Used for video-based tasks where multiple reference frames must be sampled and processed alongside a keyframe.

    Note that VideoCollect is used for video perception tasks; it is similar to MMDetection's Collect but automatically collects attributes like frame_id and is_video_data.

  7. MixFormer: End-to-End Tracking with Iterative Mixed Attention

    master
    MixFormer is a compact tracking framework built on transformers that uses a Mixed Attention Module (MAM) for simultaneous feature extraction and target information integration. It simplifies the traditional multi-stage tracking pipeline by using iterative mixed attention to extract target-specific discriminative features and perform communication between the target and the search area. The framework is built by stacking multiple MAMs with progressive patch embedding and a localization head.

  8. Understand Data Pipelines in MMTracking

    master

    MMTracking supports two types of data pipelines:

    1. Single image pipeline: Similar to MMDetection, used for standard image-based tasks. It uses VideoCollect instead of Collect to ensure compatibility with video perception tasks, collecting meta keys like frame_id and is_video_data by default.
    2. Pair-wise / multiple images pipeline: Used when multiple images (e.g., a key image and sampled reference images from the same video) must be processed simultaneously for training or inference.

  9. General settings for training and benchmarking

    master

    MMTracking follows these standard settings for training and performance evaluation:

    • Distributed Training: Enabled by default.
    • Pretrained Backbones: All PyTorch-type pretrained backbones are sourced from the official PyTorch model zoo.
    • Memory Usage: GPU memory usage is reported as the maximum value of torch.cuda.max_memory_allocated() across all 8 GPUs. This value is typically lower than what is reported by nvidia-smi.
    • Inference Speed: Benchmarking results exclude data loading time. Inference time is calculated using the tools/analysis/benchmark.py script, which computes the average time for processing 2000 images.

    Standard Benchmark Environment:

    • Hardware: 8 NVIDIA Tesla V100 (32G) GPUs, Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
    • Software: Python 3.7, PyTorch 1.5, CUDA 10.1, CUDNN 7.6.03, NCCL 2.4.08

  10. Use Flow-guided Feature Aggregation (FGFA) for Video Object Detection

    master

    FGFA is an end-to-end learning framework for video object detection that improves per-frame features by aggregating nearby features along motion paths. This method leverages temporal coherence at the feature level to improve recognition accuracy, especially for fast-moving objects in videos (e.g., those affected by motion blur or defocus).

    Pre-trained models and configurations are available for the ImageNet VID dataset using different ResNet backbones.

  12. Add a new head to SOT models

    master

    To add a custom head (the component for specific tasks like bbox prediction) to an SOT model:

    1. Define the head: Create a new file mmtrack/models/track_heads/my_head.py and register it using the @HEADS.register_module() decorator. The class should inherit from BaseModule.
    2. Import the module:
      • Option A: Add from .my_head import MyHead to mmtrack/models/track_heads/__init__.py.
      • Option B: Add custom_imports to your config file.
    3. Configure the model: Update the track_head key in your config file.
    # 1. Define the head in mmtrack/models/track_heads/my_head.py
from mmcv.runner import BaseModule
from mmdet.models import HEADS

@HEADS.register_module()
class MyHead(BaseModule):
    def __init__(self, arg1, arg2, *args, **kwargs):
        pass

    def forward(self, inputs):
        pass

# 2. Import via config (Option B)
custom_imports = dict(
    imports=['mmtrack.models.track_heads.my_head'],
    allow_failed_imports=False)

# 3. Use in config
track_head=dict(
    type='MyHead',
    arg1=xxx,
    arg2=xxx)