MONAI Label

MONAI Label is an open-source intelligent image labeling and learning platform designed for medical data. It allows users to create annotated datasets and build AI annotation models for clinical evaluation.

Key architectural concept: MONAI Label enables a serverless approach where custom labeling applications are exposed as services through the MONAI Label Server. This allows application developers to build labeling apps that can be consumed by various viewers and interfaces.

Core capabilities include:

• Automated Segmentation: Using deep learning models (e.g., UNet, UNETR) for radiology or pathology.
• Interactive Tools: Tools like DeepEdit and Deepgrow allow users to actively improve models through interaction.
• Active Learning: The system can learn from new labels online to fine-tune models, which can then be used for subsequent labeling tasks.

MONAI Label is a server-client system designed for interactive medical image annotation using AI.

Core Components:

• MONAI Label Server: A serverless way to expose custom labeling apps as services.
• Clients/Viewers: Applications that connect to the server to perform annotations.

Key Features:

• Framework for developing and deploying MONAI Label Apps (training and inference).
• Compositional and portable APIs.
• Support for various medical imaging fields including Radiology (CT/MRI), Pathology (TIFF/SVS), and Video (Endoscopy).

MONAI Label integrates with several professional radiology viewers to support visualization, segmentation, and algorithm development:

• 3D Slicer: A free, open-source software for 3D image/mesh processing. Supports radiology and monaibundle applications.
• MITK (Medical imaging Interaction ToolKit): MONAI Label is partially integrated into the MITK Workbench. It is primarily tested for inferencing using radiology and bundle apps, supporting both Auto and Click-based interactive models.
• OHIF (Open Health Imaging Foundation): A web-based platform. MONAI Label connects to the OHIF viewer via a remote DICOM server using the DICOMweb protocol.

The MONAI Label plugin supports various radiology and monaibundle applications for creating and refining labels. Examples of supported models include:

• DeepEdit
• DeepGrow
• Segmentation

Users can find more supported applications in the sample-apps/radiology/ directory of the repository.

For computer vision and video-based tasks (e.g., endoscopy), MONAI Label supports CVAT (Computer Vision Annotation Tool). This enables tasks such as:

• Endoscopy segmentation
• Object tracking

The CVAT extension supports various medical imaging tasks via the endoscopy application. These include models for segmentation, classification, and tracking. Specific application implementations can be found in the sample-apps/endoscopy/ directory of the MONAI Label repository.

Supported capabilities include:

• DeepEdit: For interactive segmentation.
• Tooltracking: For tracking surgical tools.
• InBody/OutBody classification: For body part classification.
• SAM2 Interactor: For Segment Anything Model 2 based interaction.

For digital pathology workflows, MONAI Label supports QuPath. This integration allows for the deployment of pathology-specific tasks such as:

• Nuclei segmentation
• Nuclei classification
• Tissue classification and biomarker quantification

MONAI Label integrates with various medical imaging viewers depending on the modality:

Radiology Viewers

• 3D Slicer: Most tested for radiology studies and algorithms.
• MITK: Supports inference using radiology and bundle apps (Auto and Click-based models).
• OHIF: A web-based medical imaging platform.

Pathology Viewers

• QuPath: For bioimage analysis.
• Digital Slide Archive (DSA): For managing and visualizing large imaging datasets.

Video Viewers

• CVAT: Interactive video and image annotation tool.

MONAI Label provides four main template categories:

• Radiology: Includes models for 3D image segmentation (e.g., Spleen, Liver, Kidney) using DeepEdit (interactive + auto), Deepgrow (interactive), or standard Segmentation (auto).
• Pathology: Includes models for Whole Slide Imaging (WSI) segmentation, such as Nuclei segmentation using DeepEdit or multi-label segmentation (Neoplastic, Inflammatory, etc.).
• Endoscopy: Includes tool tracking (DeepEdit or Segmentation) and classification models (InBody vs. OutBody).
• MONAI Bundle: Uses the MONAI Bundle format to pull portable deep learning models from the MONAI Model Zoo.

Note on SAM2 Support: SAM2 is included by default for these apps if python >= 3.10 is used. It provides sam_2d for single slices and sam_3d for multi-slice propagation (available in Radiology and MONAI-Bundle).

MONAI Label integrates with several open-source viewers for medical imaging:

• 3D Slicer: Most tested with radiology studies. Supports Windows and Linux. Recommended to use stable releases (>= 5.0).
• MITK: Partially integrated via MITK Workbench. Supports Auto and Click-based interactive models for radiology bundle apps.
• OHIF: A web-based platform. Allows annotation of data in a DICOM server via the MONAI Label server.
• QuPath: A platform for bioimage analysis (pathology). Supports Windows and Linux.

The Bundle App (monai bundle) empowers MONAI Label with customized models, pre- and post-processing, and specific anatomies for labeling tasks. It leverages the MONAI Bundle format, which provides:

• Self-contained model packages: All necessary information for the model is included.
• Structured configurations: Easy to override or reconstruct workflows via configuration files that separate parameter settings from Python code.
• Flexible workflows: Supports various training/inference pipelines and heterogeneous datasets (CT, MRI, Pathology, etc.).
• Model Zoo Integration: Access to hundreds of ready-to-use inference models (e.g., SwinUNETR, multi-organ abdominal segmentation) hosted in the MONAI Model Zoo.