DOCSEARCHbyAlgolia
Submit Docs

MMSegmentation Documentation

repository·main·Indexed 27 days ago

https://github.com/open-mmlab/mmsegmentation

A modular, PyTorch-based toolbox for semantic segmentation tasks within the OpenMMLab ecosystem. It provides a unified benchmark and supports various algorithms and tasks, including monocular depth estimation. The library includes implementations and configurations for models such as Asymmetric Non-local Neural Networks (ANN), Adaptive Pyramid Context Network (APCNet), BEiT, BiSeNetV1, and BiSeNetV2, with pre-trained weights available for datasets like Cityscapes, ADE20K, and Pascal VOC 2012 + Aug.

Tokens
133.5K
Snippets
367
Records
706
Agent score
93%

What's inside MMSegmentation

  1. Overview of EMANet (Expectation-Maximization Attention Networks)

    main
    EMANet is a semantic segmentation method that utilizes an Expectation-Maximization Attention (EMA) module. This module formulates the attention mechanism in an expectation-maximization manner to iteratively estimate a compact set of bases. This approach results in a low-rank representation that captures long-range relations while being more memory and computationally efficient than standard self-attention mechanisms. It is robust to input variance and includes bases maintenance and normalization methods to stabilize training.

  2. Overview of DANet (Dual Attention Network)

    main

    DANet (Dual Attention Network) is a model designed for scene segmentation that captures contextual dependencies using a self-attention mechanism. It integrates local features with global dependencies by appending two types of attention modules to a traditional dilated FCN:

    1. Position Attention Module: Selectively aggregates features at each position using a weighted sum of features from all positions, allowing similar features to be related regardless of distance.
    2. Channel Attention Module: Emphasizes interdependent channel maps by integrating associated features across all channel maps.

    The outputs of these two modules are summed to improve feature representation for more precise segmentation.

  3. Overview of BiSeNetV2 for Real-time Semantic Segmentation

    main

    BiSeNetV2 (Bilateral Segmentation Network v2) is an architecture designed for high-accuracy, high-efficiency real-time semantic segmentation. It achieves a balance between speed and accuracy by using two separate branches:

    1. Detail Branch: Uses wide channels and shallow layers to capture low-level spatial details and high-resolution features.
    2. Semantic Branch: Uses narrow channels and deep layers to obtain high-level semantic context.

    A Guided Aggregation Layer is used to fuse these two feature representations. The model is optimized for real-time performance, capable of high FPS on standard hardware (e.g., 156 FPS on a GTX 1080 Ti for 2,048x1,024 input on Cityscapes).

  4. Overview of ICNet for Real-time Semantic Segmentation

    main
    ICNet (Image Cascade Network) is designed for real-time semantic segmentation on high-resolution images. It utilizes multi-resolution branches with label guidance and a cascade feature fusion unit to reduce computation while maintaining high-quality segmentation. It is capable of real-time inference on a single GPU for datasets such as Cityscapes, CamVid, and COCO-Stuff.

  5. Overview of UNet for Biomedical Image Segmentation

    main
    UNet is a convolutional network architecture designed for biomedical image segmentation. It features a contracting path to capture context and a symmetric expanding path for precise localization. The architecture is optimized for efficient training using data augmentation, allowing it to perform well even with limited annotated training samples. It is noted for being fast, capable of segmenting a 512x512 image in less than a second on a modern GPU.

  6. Overview of NonLocal Net in MMSegmentation

    main
    NonLocal Net implements the Non-local Neural Networks architecture. It provides a generic family of building blocks designed to capture long-range dependencies by computing the response at a position as a weighted sum of features at all positions. This operation can be integrated into various computer vision architectures to improve tasks like object segmentation and pose estimation.

  7. Overview of MMSegmentation Architecture

    main

    MMSegmentation is a unified toolbox for semantic segmentation tasks, providing high-quality implementations of common methods and datasets. The framework is organized into seven core modules:

    • apis: High-level APIs for model inference.
    • structures: Defines the SegDataSample data structure for segmentation tasks.
    • datasets: Supports various semantic segmentation datasets and includes transforms for data augmentation.
    • models: The core component containing:
      • segmentors: Base classes for all segmentation models.
      • data_preprocessors: Handles input data preprocessing.
      • backbones: Feature extraction networks.
      • necks: Components connecting backbones to segmentation heads.
      • decode_heads: Predicts segmentation results from feature maps.
      • losses: Various loss functions.
    • engine: Runtime components extending MMEngine, including optimizers and hooks.
    • evaluation: Metrics for assessing model performance.
    • visualization: Tools for visualizing segmentation results.

  8. Overview of MMSegmentation

    main

    MMSegmentation is an open-source semantic segmentation toolbox built on PyTorch and part of the OpenMMLab project. It features a modular design that allows users to construct customized semantic segmentation frameworks by combining different components. Key capabilities include:

    • Unified Benchmark: A toolbox for evaluating various semantic segmentation methods.
    • Modular Design: Decomposed framework components for easy customization.
    • Out-of-the-box Support: Direct support for popular frameworks like PSPNet, DeepLabV3, PSANet, and DeepLabV3+.
    • High Efficiency: Training speeds comparable to or faster than other codebases.
    • Task Support: Beyond semantic segmentation, it supports monocular depth estimation and open-vocabulary semantic segmentation.

  9. Overview of PSPNet (Pyramid Scene Parsing Network)

    main
    PSPNet is a semantic segmentation architecture designed to exploit global context information through a pyramid pooling module. It uses different-region-based context aggregation to improve pixel-level prediction tasks. PSPNet has historically achieved state-of-the-art performance on benchmarks such as PASCAL VOC 2012 and Cityscapes.

  10. Overview of APCNet (Adaptive Pyramid Context Network)

    main
    APCNet is an Adaptive Pyramid Context Network designed for semantic segmentation. It uses Adaptive Context Modules (ACMs) to adaptively construct multi-scale contextual representations. Each ACM uses a global image representation to estimate local affinity coefficients for sub-regions, calculating a context vector based on these affinities. This approach focuses on Global-guided Local Affinity (GLA) to improve context feature construction.

  11. Overview of Asymmetric Non-local Neural Networks (ANN)

    main

    Asymmetric Non-local Neural Networks (ANN) is a semantic segmentation method designed to address the high computational and memory costs of standard non-local modules. It introduces two key components:

    1. Asymmetric Pyramid Non-local Block (APNB): Uses a pyramid sampling module to reduce computation and memory consumption while maintaining performance. For a 256x128 input, APNB is approximately 6x faster and uses 28x less GPU memory than a standard non-local block.
    2. Asymmetric Fusion Non-local Block (AFNB): An adaptation of APNB used to fuse features from different levels by considering long-range dependencies.

    This method achieved a state-of-the-art performance of 81.3 mIoU on the Cityscapes test set.

  12. Overview of PointRend

    main
    PointRend (Point-based Rendering) is a neural network module that treats image segmentation as a rendering problem. It performs point-based segmentation predictions at adaptively selected locations using an iterative subdivision algorithm. This approach allows for high-quality, crisp object boundaries and is efficient enough to support high output resolutions that are typically impractical due to memory or computational constraints. PointRend can be integrated into both instance and semantic segmentation tasks by building on top of existing state-of-the-art models.