DOCSEARCHbyAlgolia
Submit Docs

CellViT++

repository·main·Indexed 42 days ago

https://github.com/tio-ikim/cellvit-plus-plus

An energy-efficient and adaptive framework for cell segmentation and classification in pathology. It utilizes foundation models for high-performance biological image analysis and includes a cell classification module and a web-based WSI viewer.

Tokens
16.6K
Snippets
52
Records
70
Agent score
3%

What's inside CellViT++

  1. CellViT++ Framework Overview

    main

    The CellViT++ framework is composed of three primary components:

    1. CellViT++ Algorithm: The core segmentation and classification logic.
    2. Cell Classification Module: Handles the taxonomy and identification of cells.
    3. Web-Based WSI Viewer: A visualization tool for viewing Whole Slide Images (WSI).

  2. Use the Annotation Tool workflow

    main

    Once the dataset path is uploaded, the tool enters the annotation workflow:

    • Image Loading: Images and their context images are loaded from the dataset in random order.
    • Navigation:
      • The next image loads automatically after an annotation.
      • Use Skip Image to bypass an image.
      • Use Back to Last Annotation to edit your previous work.
    • Bulk Annotation: Use the Annotate All button to annotate all remaining cells within the current context image at once.
    • Exporting: Download the resulting annotations.json file from the configuration tab.

  3. Prepare a segmentation dataset for re-training

    main

    Segmentation datasets follow the same folder structure as detection datasets, but use .npy files for labels instead of .csv files. Labels must be integers starting from 1.

    Label Format (HoVer-Net style)

    Each .npy file must contain a dictionary with two keys:

    • inst_map: An instance map where each cell is a unique integer starting from 1 (0 is background). Shape: [H, W].
    • type_map: A cell class map where each integer corresponds to a cell class. Values must start from 1 (not 0).

    Refer to the example notebooks within the segmentation dataset examples for details on data loading and visualization.

  4. Set up an OpenLayers project with Vite

    main

    You can scaffold a new OpenLayers application using Vite by using the create-ol-app utility. This requires Node.js version 14 or higher.

    Follow these steps to initialize and run your development environment:

    # Scaffold the project
npx create-ol-app my-app --template vite

# Navigate to the directory
cd my-app

# Start the development server
npm start

  5. Use the NuCLS Dataset example segmentation set

    main

    The Example-Segmentation directory contains a subset of the NuCLS Dataset, specifically 10 training images and 5 test images, intended for testing segmentation tasks.

    Dataset Details:

    Citation Requirement: If you use this dataset in your work, please cite the original NuCLS paper:

    Amgad M, Atteya LA, Hussein H, Mohammed KH, Hafiz E, Elsebaie MA, Alhusseiny AM, AlMoslemany MA, Elmatboly AM, Pappalardo PA, Sakr RA. NuCLS: A scalable crowdsourcing approach and dataset for nucleus classification and segmentation in breast cancer. GigaScience. 2022 May 17;11.

  6. Prepare a dataset for the Annotation Tool

    main

    To use a custom dataset with the annotation tool, follow these requirements:

    Directory Structure:

    • The main directory must contain a config.yaml file (optional) and a folder named dataset.

    File Formats and Naming:

    • All images must be in PNG or JPG format.
    • The dataset must include cell images and optionally context images for each cell.
    • Naming Convention: The cell image and its corresponding context image must have identical filenames, with specific suffixes:
      • Cell image: [name]_roi.png
      • Context image: [name]_context.png (e.g., context.png)
    • Bounding Box Encoding: Filenames can encode bounding box coordinates for automated parsing. The format used is bb_[upper-left-xy]_bb2_[lower-right-xy].
      • Example: wsi_01_R_2_C_00_cell_9935_bb_118_120_139_137_bb2_451_365_472_382_image.png
      • bb_118_120_139_137 represents the cell image coordinates.
      • bb2_451_365_472_382 represents the cell location within the context image.
      • The name also encodes the Region ID (R_2) and Context ID (C_00).

  7. Evaluate detection-based models

    main

    To evaluate a trained model on a detection-based dataset, use the inference_cellvit_experiment_detection.py script. You must provide the correct input_shape used during training as two positional arguments (height and width).

    python3 ./cellvit/training/evaluate/inference_cellvit_experiment_detection.py \
  --logdir /path/to/your/run_log \
  --dataset_path ./path/to/dataset \
  --cellvit_path ./path/to/checkpoint.pth \
  --input_shape 256 256

    For segmentation masks (e.g., CoNSeP/HoVer-Net style), check the ./cellvit/training/evaluate folder for specific evaluation scripts.

    python3 ./cellvit/training/evaluate/inference_cellvit_experiment_detection.py \
  --logdir /path/to/your/run_log \
  --dataset_path ./test_database/training_database/Example-Detection \
  --cellvit_path ./checkpoints/CellViT-256-x40-AMP.pth \
  --input_shape 256 256

  8. Prepare a detection dataset for re-training

    main

    To train a new classifier using detection annotations, organize your data into the following folder structure:

    ├── label_map.yaml        # [optional] tracks labels
├── splits                # folder containing fold subfolders (e.g., fold_0/)
│   ├── fold_0  
│   │   ├── train.csv
│   │   └── val.csv
├── train                 # training dataset
│   ├── images            # png or jpeg files
│   └── labels            # cell annotation csv files
├── test                  # test dataset
│   ├── images
│   └── labels            # cell annotation csv files
└── train_configs         # config files (e.g., ViT256/fold_0.yaml)

    Dataset Requirements

    • Splits: Defined by CSV files in the splits folder.
    • Test Images: Must be in a dedicated test folder.
    • Image Dimensions: Must be divisible by 32 after transformations. If using non-standard dimensions, specify the input_shape (height, width) in your training configuration.
    • Annotations: CSV files containing x, y coordinates and integer labels (starting from 0).
    • Label Map: It is recommended to include a label_map.yaml to track labels.
    data:
  dataset: DetectionDataset
  dataset_path: ./path/to/dataset
  input_shape: [256, 288] # height, width

  9. Run CellViT++ using Docker

    main

    A beta Docker image is available via ikimhoerst/cellvit:beta. This image is intended for Linux/amd64 platforms and includes the CellViT-SAM-H checkpoint and all classifiers pre-installed.

    To run the container, use the following command. You must mount local directories for input slides and output results:

    docker run --name cellvit++ \
  --gpus all \
  --memory=8g \
  -v /path/to/local/input:/workspace/CellViT-plus-plus/input-data \
  -v /path/to/local/output:/workspace/CellViT-plus-plus/output-data \
  -it \
  ikimhoerst/cellvit:beta

    Notes:

    • Replace /path/to/local/input and /path/to/local/output with your actual local paths.
    • We recommend increasing --memory to 16g if you encounter issues.
    • For inference, use the same command structure as the local examples. The default model is CellViT-SAM-H, and classifiers are located in ./checkpoints/classifier/sam-h inside the container.

  10. Build and preview OpenLayers application for production

    main

    Once your development is complete, you can generate a production-ready build. The build artifacts will be located in the dist directory.

    Use the following commands to build and preview the production output:

    # Generate the production build
npm run build

# Preview the production build locally
npm run serve

  11. Train a classifier and find optimal hyperparameters

    main

    The training workflow involves running a WandB sweep to find hyperparameters, followed by retrieving the best configuration.

    1. Run WandB Sweep

    Use the --sweep flag with the training script and a sweep configuration file:

    python3 ./cellvit/train_cell_classifier_head.py --config /path/to/your/config.yaml --sweep

    2. Find Best Hyperparameter Configuration

    After the sweep, use the find_best_hyperparameter.py script to extract the best config from the sweep folder:

    python3 ./script/find_best_hyperparameter.py /path/to/your/sweep --metric AUROC/Validation

    3. Standard Training (No Sweep)

    To run training directly with a specific configuration:

    python3 ./cellvit/train_cell_classifier_head.py --config /path/to/your/config.yaml
    # Example full workflow
mkdir logs_local
python3 ./cellvit/train_cell_classifier_head.py --config ./test_database/training_database/Example-Detection/train_configs/ViT256/fold_0_sweep.yaml --sweep
python3 ./scripts/find_best_hyperparameter.py /path/to/your/sweep --metric AUROC/Validation

  12. Install CellViT++ locally via Conda and Pip (Fast Method)

    main

    The fastest way to install the environment is to use a combination of Conda for the base environment and Pip for specific packages. Follow these steps:

    1. Create the environment using the verbose YAML file:
      conda env create -f environment_verbose.yaml
    2. Activate the environment:
      conda activate cellvit_env
    3. Install the required pip packages:
      pip install -r requirements.txt
    4. Crucial: Install PyTorch specifically optimized for your system's CUDA version. Even if PyTorch was installed during the environment buildup, you should reinstall it to ensure compatibility.

    Example for CUDA 12.1:

    pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2 --index-url https://download.pytorch.org/whl/cu121
    conda env create -f environment_verbose.yaml
conda activate cellvit_env
pip install -r requirements.txt
pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2 --index-url https://download.pytorch.org/whl/cu121