Mask DINO Documentation

Mask DINO is a unified transformer-based framework designed for object detection, panoptic segmentation, instance segmentation, and semantic segmentation. It achieves task and data cooperation between detection and segmentation tasks, providing state-of-the-art performance across multiple benchmarks.

Key features include:

• A unified architecture for all major segmentation and detection tasks.
• Support for major datasets: COCO, ADE20K, and Cityscapes.
• High performance: Achieves competitive results on COCO (instance and panoptic) and ADE20K (semantic) leaderboards.

The implementation is based on detectron2 and is also available via the detrex toolbox.

MaskDINO is composed of three modular components that can be replaced with custom implementations:

1. Backbone: Define and register new backbones in maskdino/modeling/backbone (e.g., following the Swin Transformer implementation).
2. Pixel Decoder: Located in maskdino/modeling/pixel_decoder. It functions as a multi-scale encoder. It returns:
   • mask_features: Per-pixel embeddings at 1/4 resolution used for binary masks.
   • multi_scale_features: Multi-scale inputs for the Transformer decoder.
3. Transformer Decoder: Located in maskdino/modeling/transformer_decoder. It handles both detection and segmentation tasks, following the DINO decoder design.

To begin using Mask DINO, you can follow these primary paths:

• Run an Inference Demo: Use pre-trained models to see the model in action by following the guide in demo/README.md.
• Prepare Datasets: If you are training or evaluating on custom data, follow the instructions in datasets/README.md to prepare your datasets for MaskDINO.
• Explore Results: Review the performance benchmarks in the Results section of the README.
• Advanced Usage: For more complex workflows, see the More Usage section.

MaskDINO uses the DETECTRON2_DATASETS environment variable to locate builtin datasets (ADEChallengeData2016, coco, cityscapes).

• If unset, the default location is ./datasets relative to your current working directory.
• To set a custom location, use export DETECTRON2_DATASETS=/path/to/datasets.

To install MaskDINO, ensure your system meets the following requirements:

• OS: Linux
• Python: $\ge$ 3.6
• PyTorch: $\ge$ 1.9 (ensure torchvision matches your PyTorch installation and is compatible with your Detectron2 version).
• Detectron2: Must be installed following the official instructions.
• OpenCV: Optional, but required for running demos and visualization.

After meeting these requirements, install the project dependencies using: pip install -r requirements.txt

MaskDINO requires a compiled CUDA kernel for MSDeformAttn. Before compiling, ensure the CUDA_HOME environment variable is defined and points to your installed CUDA toolkit directory.

Navigate to the operations directory and run the provided shell script:

cd maskdino/modeling/pixel_decoder/ops
sh make.sh

To use COCO for instance, panoptic, or semantic segmentation, organize your files as follows:

coco/
  annotations/
    instances_{train,val}2017.json
    panoptic_{train,val}2017.json
  {train,val}2017/
    # image files mentioned in the corresponding json
  panoptic_{train,val}2017/  # png annotations
  panoptic_semseg_{train,val}2017/  # generated by the script below

Steps:

1. Install panopticapi: pip install git+https://github.com/cocodataset/panopticapi.git
2. Extract semantic annotations from panoptic annotations (required for evaluation) by running: python datasets/prepare_coco_semantic_annos_from_panoptic_annos.py

To evaluate pretrained models, download the checkpoint file and run the train_net.py script with the --eval-only flag.

Prerequisites: If your dataset files are not located within this repository, you must either set the DETECTRON2_DATASETS environment variable or create a symbolic link to your data directory inside this repo.

export DETECTRON2_DATASETS=/path/to/your/data

Command Syntax:

python train_net.py --eval-only --num-gpus <N> --config-file <config_path> MODEL.WEIGHTS <path_to_checkpoint>

Example (Instance Segmentation):

python train_net.py --eval-only --num-gpus 8 --config-file configs/coco/instance-segmentation/maskdino_R50_bs16_50ep_3s_dowsample1_2048.yaml MODEL.WEIGHTS /path/to/checkpoint_file