DOCSEARCHbyAlgolia
Submit Docs

MMSelfSup Documentation

repository·main·Indexed 25 days ago

https://github.com/open-mmlab/mmselfsup

An open-source self-supervised representation learning toolbox based on PyTorch and part of the OpenMMLab ecosystem. It provides a modular framework for implementing and benchmarking self-supervised pre-training strategies—including Barlow Twins, BEiT, BEiT v2, BYOL, CAE, DeepCluster, and DenseCL—and their application to downstream tasks such as image classification, detection, and segmentation.

Tokens
75.1K
Snippets
166
Records
367
Agent score
84%

What's inside MMSelfSup

  1. Overview of MMSelfSup Architecture

    main

    MMSelfSup is a modular open-source library for self-supervised learning (SSL). Its architecture follows the OpenMMLab design principles and consists of four main components:

    • Datasets: Supports various datasets and provides a wide range of data augmentation strategies.
    • Algorithms: Implements multiple classic self-supervised algorithms with easy-to-use user interfaces.
    • Tools: Includes training and analysis tools commonly used in self-supervised learning.
    • Benchmarks: Provides examples for using pre-trained models on various downstream tasks such as image classification, object detection, and semantic segmentation.

  2. Overview of SwAV Algorithm

    main

    SwAV (Swapped Assignments Visualizing) is an unsupervised learning algorithm for visual representations. Instead of direct pairwise feature comparisons used in contrastive learning, SwAV simultaneously clusters data and enforces consistency between cluster assignments produced by different augmentations (views) of the same image. It uses a "swapped" prediction mechanism where the code of one view is predicted from the representation of another.

    Key features include:

    • Memory Efficiency: Does not require a large memory bank or a special momentum network.
    • Multi-crop Strategy: Uses a mix of views with different resolutions to improve training without increasing compute/memory requirements.
    • Scalability: Can be trained with large or small batches and scales to unlimited data.

  3. Overview of MoCo v1 (Momentum Contrast)

    main
    MoCo v1 (Momentum Contrast) is an unsupervised visual representation learning method that treats contrastive learning as a dictionary look-up task. It utilizes a dynamic dictionary constructed via a queue and a moving-averaged encoder, allowing for a large and consistent dictionary to be built on-the-fly. This approach facilitates effective contrastive unsupervised learning and produces representations that transfer well to downstream tasks, such as ImageNet classification.

  4. Overview of MMSelfSup

    main
    MMSelfSup is an open-source self-supervised representation learning toolbox built on PyTorch and part of the OpenMMLab project. It provides state-of-the-art self-supervised learning methods with a modular design, allowing users to easily build custom algorithms. The toolbox includes standardized benchmarks for tasks such as logistic regression, SVM, semi-supervised classification, object detection, and semantic segmentation. It is designed for compatibility with other OpenMMLab projects for smooth evaluation on downstream tasks.

  5. Overview of BYOL (Bootstrap Your Own Latent)

    main
    BYOL is a self-supervised image representation learning approach that uses two neural networks: an online network and a target network. The online network is trained to predict the target network's representation of the same image under a different augmented view. The target network is updated using a slow-moving average of the online network's weights.

  6. Overview of DeepCluster

    main
    DeepCluster is an unsupervised learning method for visual features that jointly learns neural network parameters and cluster assignments. It works by iteratively grouping features using the k-means clustering algorithm and using those assignments as supervision to update the network weights. This implementation is based on the paper Deep Clustering for Unsupervised Learning of Visual Features.

  7. Overview of Online Deep Clustering (ODC)

    main

    Online Deep Clustering (ODC) is an unsupervised representation learning method that performs clustering and network updates simultaneously. It uses two dynamic memory modules to stabilize training:

    1. Samples memory: Stores samples' labels and features.
    2. Centroids memory: Manages centroid evolution.

    This approach avoids the instability of alternating between feature clustering and network parameter updates by integrating label re-assignment and memory updates directly into the network update iterations.

  8. Overview of DenseCL

    main
    DenseCL (Dense Contrastive Learning) is a self-supervised visual pre-training method designed to bridge the gap between image-level and pixel-level predictions. Unlike standard self-supervised methods optimized for classification, DenseCL optimizes a pairwise contrastive (dis)similarity loss at the pixel level (local features) by considering the correspondence between local features in two views of an input image. This makes it particularly effective for dense prediction tasks like detection and segmentation.

  9. Overview of MaskFeat (Masked Feature Prediction)

    main
    MaskFeat is a self-supervised visual pre-training method that masks portions of an input sequence and predicts the features of the masked regions. It is designed for both video and image models. A key finding is that using Histograms of Oriented Gradients (HOG) as the target feature descriptor provides high performance and efficiency. MaskFeat can be used to pre-train large-scale Transformer-based models on unlabeled video or image data.

  10. Overview of BEiT v2

    main
    BEiT v2 (Masked Image Modeling with Vector-Quantized Visual Tokenizers) is a self-supervised representation learning method that uses a semantic-rich visual tokenizer as the reconstruction target for masked prediction. It promotes Masked Image Modeling (MIM) from pixel-level to semantic-level using vector-quantized knowledge distillation. It is effective for tasks like image classification and semantic segmentation.

  11. Overview of MMSelfSup transforms

    main

    MMSelfSup organizes transforms into three main categories within a Pipeline:

    1. Data Processing Transforms: Used for image augmentation and manipulation. Examples include RandomCrop, RandomResizedCrop, and RandomGaussianBlur. Some transforms may be imported from other repositories (e.g., LoadImageFromFile from MMCV).
    2. Multi-view Wrappers: Transforms that wrap multiple views of an image for algorithms requiring multi-view inputs.
    3. Formatting Transforms: Transforms used to pack data into a format compatible with specific algorithm inputs, typically placed at the end of a pipeline.

  12. Overview of MILAN (Masked Image Pretraining on Language Assisted Representation)

    main

    MILAN is a masked image pretraining method that reconstructs image features using semantic signals from caption supervision, rather than predicting raw pixels. It utilizes a prompting decoder architecture and a semantic-aware mask sampling mechanism to improve transfer performance.

    Key performance benchmarks (on ImageNet-1K with ViT-B/16 backbone):

    • Pretraining: 400 epochs, batch size 4096.
    • Linear Evaluation: 78.9% Top-1 accuracy.
    • Fine-tuning: 85.3% Top-1 accuracy.
    • Semantic Segmentation (ADE20K): 52.7 mIoU.