MaskFormer

MaskFormer provides a unified architecture for semantic- and instance-level segmentation tasks.

Key Features:

• Unified Segmentation: Provides a single view for both semantic and instance-level segmentation.
• Efficiency: Achieves better results with improved efficiency.
• Dataset Support: Supports major semantic segmentation datasets including ADE20K, Cityscapes, COCO-Stuff, and Mapillary Vistas.
• Detectron2 Integration: Supports ALL Detectron2 models.

Note on Mask2Former: For a more advanced universal architecture that achieves SOTA on panoptic, instance, and semantic segmentation, users should check out Mask2Former.

You can use demo.py to run inference on images, videos, or a webcam using pre-trained weights. Because the provided configuration files are designed for training, you must explicitly provide the path to a checkpoint file using the MODEL.WEIGHTS option.

Common arguments for demo.py:

• --input <files>: Specify one or more image files.
• --webcam: Use the computer's webcam as input.
• --video-input <video_path>: Use a video file as input.
• --output <path>: Save outputs to a directory (for images) or a file (for webcam/video).
• --opts MODEL.DEVICE cpu: Run inference on the CPU instead of GPU.
• --opts MODEL.WEIGHTS <path>: Specify the path to the model checkpoint.

cd demo/
python demo.py --config-file ../configs/ade20k-150/maskformer_R50_bs16_160k.yaml \
  --input input1.jpg input2.jpg \
  --opts MODEL.WEIGHTS /path/to/checkpoint_file

To prepare ADE20K for panoptic segmentation, you must combine semantic and instance annotations.

1. Install panopticapi:

pip install git+https://github.com/cocodataset/panopticapi.git

2. Download instance annotations:

wget http://sceneparsing.csail.mit.edu/data/ChallengeData2017/annotations_instance.tar

3. Run the preparation script:

python datasets/prepare_ade20k_pan_seg.py

Expected Structure:

ADEChallengeData2016/
  images/
  annotations/
  objectInfo150.txt
  annotations_instance/
  annotations_detectron2/
  ade20k_panoptic_train.json
  ade20k_panoptic_train/
  ade20k_panoptic_val.json
  ade20k_panoptic_val/

To begin using MaskFormer, follow these two primary steps:

1. Prepare your datasets: Follow the guide in datasets/README.md to format your data correctly.
2. Run MaskFormer: Follow the step-by-step guide in GETTING_STARTED.md to execute the model.

To use ADE20k Scene Parsing, ensure your directory structure matches the expected format. The annotations_detectron2 directory must be generated using the provided script.

Expected Structure:

ADEChallengeData2016/
  annotations/
  annotations_detectron2/
  images/
  objectInfo150.txt

Generation Command:

python datasets/prepare_ade20k_sem_seg.py

To evaluate a model's performance without starting a training run, use train_net.py with the --eval-only flag and provide the path to your checkpoint via MODEL.WEIGHTS.

./train_net.py \
  --config-file configs/ade20k-150/maskformer_R50_bs16_160k.yaml \
  --eval-only MODEL.WEIGHTS /path/to/checkpoint_file

MaskFormer commonly initializes from backbone models pre-trained on ImageNet. The following Detectron2 pretrained models are available:

• R-50.pkl (torchvision): A converted copy of torchvision's ResNet-50. Conversion details are in tools/convert-torchvision-to-d2.py.
• R-103.pkl: A ResNet-101 where the first 7x7 convolution is replaced by three 3x3 convolutions (also known as ResNet101c). Pre-trained on ImageNet using standard PyTorch recipes.
• R-50.pkl (MSRA): Converted copy of MSRA's original ResNet-50.
• R-101.pkl (MSRA): Converted copy of MSRA's original ResNet-101.
• X-101-32x8d.pkl: ResNeXt-101-32x8d model trained with Caffe2 at FB.

To install MaskFormer, ensure your environment meets the following requirements and then install the dependencies via pip.

Requirements

• OS: Linux or macOS
• Python: $\ge$ 3.6
• PyTorch: $\ge$ 1.7 and a matching torchvision installation. It is recommended to install these together from pytorch.org. Ensure the PyTorch version is compatible with your Detectron2 installation.
• Detectron2: Must be installed following the official Detectron2 installation instructions.
• OpenCV: Optional, but required if you intend to run demos or use visualization features.

Installation Steps

After satisfying the requirements above, install the project dependencies using:

pip install -r requirements.txt

To reproduce any model listed in the Model Zoo, use the provided configuration file with the train_net.py script.

Note on Hardware Requirements:

• Most models were trained using 8 NVIDIA V100 GPUs with NVLink.
• COCO panoptic segmentation models require 64 NVIDIA V100 GPUs with distributed training to reproduce the reported results.

To use COCO-Stuff-10K v1.0, download the annotations and run the preparation script.

1. Download annotations:

wget http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/cocostuff-10k-v1.0.zip

2. Setup structure: Unzip the file and place annotations, imageLists, and images into the coco/coco_stuff_10k/ directory.

3. Generate Detectron2 files:

python datasets/prepare_coco_stuff_10k_v1.0_sem_seg.py

Expected Structure:

coco/
  coco_stuff_10k/
    annotations/
    imageLists/
    images/
    annotations_detectron2/
      train/
      test/
    images_detectron2/
      train/
      test/