DOCSEARCHbyAlgolia
Submit Docs

Mask2Former

repository·main·Indexed 25 days ago

https://github.com/facebookresearch/mask2former

A universal image segmentation architecture supporting panoptic, instance, and semantic segmentation. It works with datasets including ADE20K, Cityscapes, COCO, and Mapillary Vistas. The project includes tools for model training, evaluation, and inference, as well as guides for replacing backbones, pixel decoders, and Transformer decoders. It requires PyTorch (≥ 1.9), Detectron2, and a compiled CUDA kernel for MSDeformAttn.

Tokens
3.6K
Snippets
11
Records
39
Agent score
86%

What's inside Mask2Former

  1. Run an inference demo with pre-trained models

    main

    Use demo.py to run inference and visualize results using OpenCV. You must provide a configuration file and specify the model weights using the MODEL.WEIGHTS option, as the default configs are designed for training.

    Common arguments:

    • --webcam: Run inference on your webcam.
    • --video-input <path>: Run inference on a video file.
    • --output <path>: Save outputs to a directory (for images) or a file (for webcam/video).
    • --opts MODEL.DEVICE cpu: Run inference on the CPU instead of GPU.
    cd demo/
python demo.py --config-file ../configs/coco/panoptic-segmentation/maskformer2_R50_bs16_50ep.yaml \
  --input input1.jpg input2.jpg \
  --opts MODEL.WEIGHTS /path/to/checkpoint_file

  2. Convert torchvision weights to D2 format

    main

    Use convert-torchvision-to-d2.py to convert pre-trained torchvision weights into a format compatible with Detectron2 (D2).

    wget https://download.pytorch.org/models/resnet101-63fe2227.pth
python tools/convert-torchvision-to-d2.py resnet101-63fe2227.pth R-101.pkl

  3. Analyze model parameters and FLOPs

    main

    Use analyze_model.py to calculate model parameters and FLOPs. The method of calculation depends on the task and dataset.

    Semantic Segmentation (ADE20K only)

    Warning: Use with caution. This uses a dummy image with a fixed size equal to cfg.INPUT.CROP.SIZE[0] x cfg.INPUT.CROP.SIZE[0]. Do not use --use-fixed-input-size for other datasets like Cityscapes.

    python tools/analyze_model.py --num-inputs 1 --tasks flop --use-fixed-input-size --config-file CONFIG_FILE

    Panoptic and Instance Segmentation

    This computes the average FLOPs over 100 real validation images.

    python tools/analyze_model.py --num-inputs 100 --tasks flop --config-file CONFIG_FILE

  5. Replace the backbone in Mask2Former

    main
    To use a custom backbone, you must define and register it within the mask2former/modeling/backbone directory. You can use the existing Swin Transformer implementation as a reference for the required structure. After registration, update your configuration file to point to your new backbone name.

  6. Prepare Mapillary Vistas for instance segmentation evaluation

    main

    While no preprocessing is needed for semantic and panoptic segmentation on Mapillary Vistas, you must run the following script to generate COCO-style instance annotations if you intend to evaluate instance segmentation.

    python datasets/prepare_mapillary_vistas_ins_seg.py

  7. Replace the pixel decoder in Mask2Former

    main

    To use a custom pixel decoder, define and register it under mask2former/modeling/pixel_decoder and update your configuration file.

    Your implementation must include a self.forward_features(features) method that returns a tuple of three values:

    1. mask_features: Per-pixel embeddings with a resolution of 1/4 of the original image (used for binary mask production).
    2. None: The second return value can be None.
    3. multi_scale_features: A list of length 3 containing multi-scale inputs for the Transformer decoder (e.g., resolutions 1/32, 1/16, and 1/8).
    MODEL:
  SEM_SEG_HEAD:
    # pixel decoder
    PIXEL_DECODER_NAME: "TransformerEncoderPixelDecoder"
    IN_FEATURES: ["res2", "res3", "res4", "res5"]
    COMMON_STRIDE: 4
    TRANSFORMER_ENC_LAYERS: 6

  8. Train Mask2Former models via command line

    main

    Use train_net.py to train models. Ensure you have set up your datasets according to the datasets/README.md instructions first.

    Multi-GPU Training: By default, configs are optimized for 8-GPU training. Use the --num-gpus flag to specify the count.

    Single-GPU Training: When training on a single GPU, you must manually adjust the batch size (SOLVER.IMS_PER_BATCH) and the learning rate (SOLVER.BASE_LR) to account for the change in scale.

  9. Install Mask2Former requirements

    main
    To install Mask2Former, ensure you are on Linux or macOS with Python ≥ 3.6. You must install PyTorch (≥ 1.9) and a matching torchvision version. Detectron2 is a required dependency. OpenCV is optional but required if you intend to use the demo or visualization features. Finally, install the project dependencies using pip install -r requirements.txt.

  10. Complete Mask2Former setup via Conda

    main

    This guide provides a full sequence for setting up a dedicated Conda environment, including PyTorch, OpenCV, Detectron2, and the necessary dataset APIs (COCO Panoptic and Cityscapes).

    conda create --name mask2former python=3.8 -y
conda activate mask2former
conda install pytorch==1.9.0 torchvision==0.10.0 cudatoolkit=11.1 -c pytorch -c nvidia
pip install -U opencv-python

# under your working directory
git clone git@github.com:facebookresearch/detectron2.git
cd detectron2
pip install -e .
pip install git+https://github.com/cocodataset/panopticapi.git
pip install git+https://github.com/mcordts/cityscapesScripts.git

cd ..
git clone git@github.com:facebookresearch/Mask2Former.git
cd Mask2Former
pip install -r requirements.txt
cd mask2former/modeling/pixel_decoder/ops
sh make.sh

  11. Build Mask2Former on a system without a GPU (drivers only)

    main

    If you need to build the project on a system that has CUDA drivers but lacks a physical GPU device, use the following command to force CUDA compilation for a specific architecture (e.g., '8.0'):

    TORCH_CUDA_ARCH_LIST='8.0' FORCE_CUDA=1 python setup.py build install

  12. Prepare COCO dataset for panoptic evaluation

    main

    To use COCO for panoptic segmentation, ensure your directory follows the expected structure (including annotations/ and panoptic_{train,val}2017/ folders). You must install panopticapi and run the provided script to extract semantic annotations from panoptic annotations, which is required for evaluation.

    pip install git+https://github.com/cocodataset/panopticapi.git
python datasets/prepare_coco_semantic_annos_from_panoptic_annos.py