DOCSEARCHbyAlgolia
Submit Docs

MMPreTrain Documentation

repository·main·Indexed 26 days ago

https://github.com/open-mmlab/mmpretrain

An open-source pre-training toolbox based on PyTorch and part of the OpenMMLab project. MMPreTrain provides a comprehensive suite of backbones, training strategies (supervised, self-supervised, multi-modal), and inference tasks for computer vision, including support for models such as ArcFace, Barlow Twins, BEiT, BEiTv2, BLIP, and BLIP-2.

Tokens
133.6K
Snippets
470
Records
694
Agent score
37%

What's inside MMPreTrain

  1. Overview of MMPretrain

    main
    MMPretrain is an open-source framework for computer vision pre-training. It provides multiple powerful pre-trained backbones and supports various pre-training strategies. The project originated from MMClassification and MMSelfSup and is designed to streamline research and engineering for vision recognition tasks by providing rich pre-trained models for downstream tasks.

  2. Overview of ResNeSt (Split-Attention Networks)

    main
    ResNeSt (Split-Attention Networks) is a modularized architecture designed for visual recognition. It applies channel-wise attention across different network branches to capture cross-feature interactions and learn diverse representations. ResNeSt is designed to outperform EfficientNet in the accuracy and latency trade-off for image classification and serves as a highly effective backbone for transfer learning tasks.

  3. Overview of MMPreTrain features

    main

    MMPreTrain is an open-source pre-training toolbox based on PyTorch, part of the OpenMMLab project. It supports PyTorch 1.8+ and provides:

    • Various backbones and pretrained models.
    • Rich training strategies: Supervised learning, self-supervised learning (e.g., MAE, BEiT, iTPN, SparK), and multi-modality learning.
    • Out-of-the-box inference tasks:
      • Image Classification
      • Image Caption
      • Visual Question Answering
      • Visual Grounding
      • Retrieval (Image-To-Image, Text-To-Image, Image-To-Text)
    • High efficiency and extensibility with large-scale training configs and model analysis toolkits.

  4. Understand the Design of Data Pipelines

    main

    In MMPretrain, a data pipeline is a sequence of data transforms used to process a sample dictionary when indexing a sample from a dataset. Each transform takes a dictionary as input, processes it, and outputs a dictionary for the next transform in the sequence.

    Commonly, datasets only load image paths and labels initially to save memory, and the data pipeline handles the actual loading of image content and subsequent augmentations during training or testing.

  6. Explore the MMPreTrain Model Zoo

    main

    MMPreTrain provides a wide range of pre-trained models and algorithms categorized into several domains:

    • Backbones: Includes architectures like ResNet, Swin-Transformer, Vision-Transformer, ConvNeXt, MobileNet V2/V3, and many others.
    • Self-Supervised Learning: Supports algorithms such as MoCo (V1, V2, V3), SimCLR, BYOL, SwAV, MAE, and BEiT.
    • Multi-modal Algorithms: Includes BLIP, BLIP-2, OFA, Flamingo, Chinese CLIP, MiniGPT-4, and LLaVA.
    • Image Retrieval: Supports ArcFace.

    You can find detailed statistics and specific model configurations in the official Model Zoo.

  7. Use mmpretrain.engine runtime components

    main
    The mmpretrain.engine package provides specialized runtime components for classification tasks, including hooks, runners, optimizers, and loops. Note that some of these components are specific to mmpretrain and may not be supported by MMEngine currently, though they may be migrated in the future.

  8. Understand the MMPretrain Model Architecture

    main

    MMPretrain models are composed of several functional modules. A complete model is a top-level module that orchestrates these components:

    • model: The top-level module defining the task type (e.g., ImageClassifier, MAE, ImageToImageRetriever).
    • backbone: The feature extraction network (e.g., ResNet, MobileNet).
    • neck: The component between the backbone and the head (e.g., GlobalAveragePooling).
    • head: The component for task-specific operations (e.g., ClsHead, ContrastiveHead).
    • loss: The component within the head used for calculating losses (e.g., CrossEntropyLoss, LabelSmoothLoss).
    • target_generator: Specifically for self-supervised learning tasks (e.g., VQKD, HOGGenerator).

  9. Explore MMPretrain Model Zoo and Analysis Tools

    main

    MMPretrain provides several resources for model selection and analysis:

    • Model Zoo: Browse the model zoo for summaries of backbones, pre-training algorithms, and specific papers.
    • Visualization Tools: Use tools for dataset visualization, scheduler visualization, CAM (Class Activation Map) visualization, and t-SNE visualization.
    • Analysis Tools: Perform model analysis using tools for printing configs, verifying datasets, log result analysis, complexity analysis, confusion matrices, and shape bias analysis.
    • Deployment: Refer to the model serving guide for deployment instructions.

  11. Getting Started with MMPretrain

    main

    MMPretrain is an open-source pre-training algorithm framework providing powerful pre-trained backbones and various pre-training strategies. It is evolved from MMClassification and MMSelfSup.

    To get started, follow these recommended paths:

    1. Environment Setup: Read the get_started.md guide to set up your environment.
    2. Basic Usage: Follow the tutorials to learn how to obtain pre-trained models and evaluate them on downstream tasks. Key topics include:
      • Configuration (user_guides/config.md)
      • Dataset Preparation (user_guides/dataset_prepare.md)
      • Inference (user_guides/inference.md)
      • Training (user_guides/train.md)
      • Testing (user_guides/test.md)
      • Downstream Tasks (user_guides/downstream.md)
    3. Customization: For algorithm customization, refer to the Advanced Guides covering datasets, pipelines, modules, schedules, runtime, evaluation, and conventions.

  12. Plot shape bias results

    main

    After inference, a CSV file named cue-conflict_model-name_session-1.csv will be generated in your csv_dir.

    Important: You must also download the original cue-conflict CSV files and place them in the same csv_dir before plotting.

    Use tools/analysis_tools/shape_bias.py to generate the final plot.

    python tools/analysis_tools/shape_bias.py --csv-dir $CSV_DIR --result-dir $RESULT_DIR --colors $RGB --markers o --plotting-names $YOUR_MODEL_NAME --model-names $YOUR_MODEL_NAME