Matterport Mask R-CNN Implementation

Use the nucleus.py script to train models for the 2018 Data Science Bowl challenge (segmenting nuclei in microscopy images). You can train starting from ImageNet weights, from a specific weights file, or resume training from a previous session.

Training from ImageNet weights

Use the train subset (which is stage1_train minus the validation set):

python3 nucleus.py train --dataset=/path/to/dataset --subset=train --weights=imagenet

Training from specific weights

Use the full stage1_train dataset and a specific .h5 weights file:

python3 nucleus.py train --dataset=/path/to/dataset --subset=stage1_train --weights=/path/to/weights.h5

Resuming training

To resume training from the last saved checkpoint:

python3 nucleus.py train --dataset=/path/to/dataset --subset=train --weights=last

To install the Mask R-CNN implementation, follow these steps:

1. Clone the repository.
2. Install the required dependencies:

   pip3 install -r requirements.txt

3. Run the setup from the repository root:

   python3 setup.py install

4. Download the pre-trained COCO weights (mask_rcnn_coco.h5) from the project's releases page.

Requirements:

• Python 3.4
• TensorFlow 1.3
• Keras 2.0.8

MS COCO specific requirements: If you intend to train or test on MS COCO, you must also install pycocotools. Use a Python 3 compatible fork:

• Linux: https://github.com/waleedka/coco
• Windows: https://github.com/philferriere/cocoapi (requires Visual C++ 2015 build tools).

To use the Balloon color splash example, you must download specific assets and place them in the correct directory structure:

1. Weights: Download mask_rcnn_balloon.h5 from the Releases page and save it in the root mask_rcnn directory.
2. Dataset: Download balloon_dataset.zip and extract it so that the contents are located at mask_rcnn/datasets/balloon/.

To explore the dataset or the detection pipeline step-by-step, you can use the provided Jupyter notebooks: inspect_balloon_data.ipynb or inspect_balloon_model.ipynb.

The Nuclei sample includes two Jupyter notebooks for exploring the pipeline:

• inspect_nucleus_data.ipynb: Used to explore the dataset and run statistical analysis.
• inspect_nucleus_model.ipynb: Used to walk through the detection process step-by-step.

These notebooks are useful for debugging and visualizing how the model segments nuclei in microscopy images.

The repository provides several Jupyter notebooks to visualize the detection pipeline at various stages for debugging and understanding:

• samples/coco/inspect_data.ipynb: Visualizes pre-processing steps for training data.
• samples/coco/inspect_model.ipynb: Provides deep inspection of the detection and segmentation pipeline, including:
  • Anchor sorting and filtering: Visualizes the Region Proposal Network (RPN) stages.
  • Bounding Box Refinement: Shows final detection boxes vs. applied refinements.
  • Mask Generation: Visualizes generated masks before scaling/placement.
  • Layer activations: Inspects activations to find issues like zeroed or noisy layers.
• samples/coco/inspect_weights.ipynb: Inspects trained model weights and looks for anomalies via weight histograms.

To train the model on a custom dataset, you need to extend two primary classes:

1. Config: Subclass this to modify default configuration attributes (e.g., learning rate, batch size, image resizing).
2. Dataset: Subclass this to provide a consistent way to load your data. This allows the model to work with new datasets without changing the core implementation and supports loading multiple datasets simultaneously.

Reference implementations for custom datasets can be found in:

• samples/shapes/train_shapes.ipynb (Toy dataset)
• samples/coco/coco.py
• samples/balloon/balloon.py
• samples/nucleus/nucleus.py

You can train a model specifically for balloon detection using balloon.py. You can start training from different weight initializations or resume an existing training session.

Training options:

• From COCO weights: Use --weights=coco.
• From ImageNet weights: Use --weights=imagenet.
• Resume training: Use --weights=last to continue from the last saved checkpoint.

Note on configuration: The default configuration in balloon.py is set to train for 3,000 steps (30 epochs of 100 steps each) with a batch size of 2. You should update the training schedule in the script to suit your specific requirements.

To use the Nucleus dataset, download it from the Kaggle Data Science Bowl 2018 competition and save it to mask_rcnn/datasets/nucleus. Use nucleus.NucleusDataset() to initialize the dataset object and load_nucleus() to load specific subsets.

Supported subsets:

• train: Loads stage1_train but excludes validation images.
• val: Loads validation images from stage1_train.
• Specific stage names (e.g., stage1_test).

You must call dataset.prepare() after loading to initialize the dataset metadata.

# Load dataset
dataset = nucleus.NucleusDataset()
# The subset is the name of the sub-directory, such as stage1_train, stage1_test, ...etc.
dataset.load_nucleus(DATASET_DIR, subset="train")

# Must call before using the dataset
dataset.prepare()

To improve training speed and reduce memory usage, Mask R-CNN can use 'mini masks'. This involves:

1. Storing mask pixels only within the object's bounding box rather than the full image.
2. Resizing the mask to a smaller fixed size (e.g., 56x56).

You can enable this in modellib.load_image_gt by setting use_mini_mask=True. To visualize the effect, you can use utils.expand_mask to map the mini mask back to the original image dimensions.

# Add augmentation and mask resizing.
image, image_meta, class_ids, bbox, mask = modellib.load_image_gt(
    dataset, config, image_id, augment=True, use_mini_mask=True)

# Expand mask back to original image shape for visualization
mask = utils.expand_mask(bbox, mask, image.shape)
visualize.display_instances(image, bbox, mask, class_ids, dataset.class_names)