DOCSEARCHbyAlgolia
Submit Docs

DeepLIIF (Deep-Learning Inferred Multiplex Immunofluorescence)

repository·main·Indexed 42 days ago

https://github.com/nadeemlab/deepliif

A multitask deep learning framework for immunohistochemical (IHC) image quantification. DeepLIIF performs stain deconvolution, cell segmentation, and quantitative single-cell IHC scoring in a single step to translate IHC slides into multiplex immunofluorescence (mpIF) modalities. It supports deployment via Docker, Torchserve, and a cloud-native web interface, and includes an ImageJ plugin and a Python CLI for training and testing.

Tokens
16.3K
Snippets
49
Records
79
Agent score
54%

What's inside DeepLIIF

  1. Overview of DeepLIIF

    main

    DeepLIIF (Deep-Learning Inferred Multiplex Immunofluorescence) is a multitask deep learning framework designed for IHC (immunohistochemical) image quantification. It provides a single-step solution for:

    1. Stain Deconvolution/Separation: Translating low-cost IHC slides into high-information multiplex immunofluorescence (mpIF) modalities (e.g., Hematoxylin, DAPI, and specific protein channels).
    2. Cell Segmentation: Performing cell delineation using a nuclear-envelope stain (LAP2beta) to improve accuracy.
    3. Quantitative Single-Cell IHC Scoring: Classifying cells as positive (red in masks) or negative (blue in masks) based on protein expression.

    The model is designed to generalize from clean training data (e.g., Ki67) to noisier images and various other markers such as CD3, CD8, BCL2, BCL6, MYC, MUM1, CD10, and TP53.

  2. Adjust Hyper-parameters when moving to Multi-GPU

    main

    When moving from single-GPU to multi-GPU training, your effective batch size increases. This affects how gradients are applied and how often weight updates occur.

    Batch Size & Learning Rate Relationship:

    • In DDP: Gradients are averaged across all processes. This results in fewer gradient descent steps per epoch. To compensate for the larger effective batch size, you should increase the learning rate proportionally.
    • In DP: Gradients are summed across GPUs. Performance/accuracy may still suffer from the larger effective batch size, so increasing the learning rate is also recommended here.

  3. Multi-GPU Training: DP vs DDP

    main

    DeepLIIF supports two modes for multi-GPU training depending on your hardware setup:

    1. Data Parallel (DP): Triggered by deepliif train. This is a single-process method. All GPUs must be on the same machine.
    2. Distributed Data Parallel (DDP): Triggered by deepliif trainlaunch. This spawns multiple processes and can be used across multiple machines.

    Example for 2 GPUs:

    # DP (Single machine)
deepliif train --dataroot <data_dir> --batch-size 6 --gpu-ids 0 --gpu-ids 1

# DDP (Multi-machine/Multi-process)
deepliif trainlaunch --dataroot <data_dir> --batch-size 3 --gpu-ids 0 --gpu-ids 1 --use-torchrun "--nproc_per_node 2"

  4. How synthetic IHC image generation works

    main

    DeepLIIF uses a GAN-based model to generate synthetic IHC images from co-registered data to improve the model's ability to handle clustered positive cells.

    The Generation Process:

    • Inputs: The model receives a Hematoxylin channel, a multiplex immunofluorescence (mpIF) DAPI image, and a segmentation mask.
    • Hematoxylin Channel: Converted to grayscale to provide texture and shape information for the background.
    • DAPI Image: Assists in identifying the location, shape, and texture of cells to isolate them from the background.
    • Segmentation Mask: Specifies cell color based on type (e.g., brown hue for positive cells, blue hue for negative cells).

    Increasing Cluster Density: To create more clustered positive cells, the 'Neg-to-Pos' parameter is used to convert a percentage of random negative cells in the segmentation mask into positive cells during the synthesis process.

  5. Adjust Hyper-parameters for Multi-GPU training

    main

    When moving from single-GPU to multi-GPU training, the effective batch size increases. To maintain similar training performance, you should consider:

    • Learning Rate: Increase the learning rate proportionally to the increase in the effective batch size. This compensates for the fact that DDP performs fewer gradient descent steps per epoch (as it averages gradients across processes).
    • Batch Size: Be aware that in DDP, the effective batch size is the sum of the batch sizes of all processes.

  6. How DeepLIIF uses Dask for deployment

    main

    DeepLIIF networks are deployed using a combination of TorchScript and Dask.

    • TorchScript is used to serialize and optimize models from pre-trained checkpoints and Python code.
    • Dask is used to express model parallelization and interdependencies via Dask [Delayed] functions, allowing for the construction of a computational graph.

    The concrete implementation of the Dask logic is located in the run_dask() function within the deepliif.models module.

  7. Deploy DeepLIIF using Torchserve

    main

    DeepLIIF can be deployed using Torchserve workflows. The inference process follows a 4-stage DAG (Directed Acyclic Graph):

    1. Pre-process: Deserializes the image and returns a tensor.
    2. G1-4: Runs ResNets to generate Hematoxylin, DAPI, LAP2, and Ki67 masks.
    3. G51-5: Runs UNets and applies Weighted Average to generate the Segmentation image.
    4. Aggregate: Aggregates and serializes results for the user.

    Typical Workflow:

    1. Load IHC image and generate tiles.
    2. For each tile:
      • Resize to 512x512 and transform to tensor.
      • Serialize the tensor and call the inference API.
      • Deserialize the results.
    3. Stitch results back together and apply post-processing.

  8. Choose between Data Parallel (DP) and Distributed Data Parallel (DDP)

    main

    DeepLIIF supports two methods for multi-GPU training via PyTorch:

    Data Parallel (DP)

    • Mechanism: Single-process, multi-threading.
    • Constraint: All GPUs must be on the same physical machine.
    • Best for: Users familiar with previous DeepLIIF releases, users without multi-GPU machines, or those who do not mind slower training speeds due to multi-threading overhead.
    • Batch Size: The --batch-size you set is the effective batch size (per process).

    Distributed Data Parallel (DDP)

    • Mechanism: Multiple processes (typically 1 process per GPU).
    • Constraint: Can be distributed across multiple machines (nodes).
    • Best for: Maximum training speed and cross-node distribution.
    • Batch Size: The --batch-size is per process. The effective batch size is batch-size * number of processes.
    • Note: DDP requires extra GPU memory for CUDA communication buffers, which might result in a smaller effective batch size compared to DP in some scenarios.

  9. Prepare training data with `prepare-training-data`

    main

    To train DeepLIIF, image sets must be 512x512 and combined into 3072x512 horizontal strips (six 512x512 images stitched together).

    Use the prepare-training-data CLI command to automate this. The input directory must contain image sets where each set includes six specific image types with matching names, differing only by their label.

    Required Label Names:

    • IHC
    • Hematoxylin
    • DAPI
    • Lap2
    • Marker
    • Seg

    The command creates train and val directories within the specified --output-dir and distributes images based on the --validation-ratio.

    deepiliif prepare-training-data --input-dir /path/to/input/images \
                               --output-dir /path/to/output/images \
                               --validation-ratio 0.2

  10. Perform inference on images

    main

    Run inference on input images using either the CLI or the test.py script.

    Usage Notes:

    • Input Formats: deepliif test expects only input modalities (e.g., IHC for original DeepLIIF). test.py expects the same format as training/validation data.
    • Tile Size: You must specify --tile-size. Recommended values based on magnification:
      • 40x: 512
      • 20x: 256
      • 10x: 128
    • Eager Mode: Use --eager-mode to run original model files directly without serialization.
    • Optimization: Use --seg-only to reduce runtime if you only need final segmentations and not the inferred multiplex images.
    # Using the CLI
deepliif test --input-dir /path/to/input/images --output-dir /path/to/output/images --model-dir /path/to/the/serialized/model --tile-size 512

# Using the python script
python test.py --dataroot /path/to/input/images --results_dir /path/to/output/images --checkpoints_dir /path/to/model/files --name Model_Name

  11. Submit multiple ROIs to DeepLIIF

    main

    Use this workflow to process multiple specific regions of interest at once.

    Constraints: Individual ROI dimensions must not exceed 3000 x 3000 pixels.

    1. Open an image file in ImageJ.
    2. Create your desired regions of interest and add them to the ImageJ ROI Manager.
    3. Navigate to Plugins > DeepLIIF > Submit ROIs to DeepLIIF.
    4. Select the appropriate resolution/magnification: 10x, 20x, or 40x.
    5. Click OK and wait for processing.

    Outputs:

    • Inferred ROI images and IHC scoring are stored in folders in the same directory as the original image.
    • The classification overlay regions and combined IHC score are displayed in ImageJ.