DOCSEARCHbyAlgolia
Submit Docs

GigaTIME Documentation

repository·main·Indexed 18 days ago

https://github.com/prov-gigatime/gigatime

A multimodal AI framework for generating virtual populations for tumor microenvironment modeling by predicting spatial proteomics (mIF) from routine H&E pathology slides. The repository includes the original CNN-based GigaTIME model and GigaTIME-Flash, an optimized version using a DINOv2-small ViT encoder with LoRA adapters and a convolutional decoder for faster inference and lower GPU memory usage. It supports whole-slide image (WSI) processing for TCGA slides via openslide or tiffslide.

Tokens
4.7K
Snippets
14
Records
22
Agent score
63%

What's inside GigaTIME

  1. GigaTIME Intended Use and Limitations

    main

    Primary Intended Use

    Support AI researchers in reproducing and building upon research. GigaTIME is designed to generate virtual mIF (multiplex immunofluorescence) profiles from routine H&E pathology slides.

    Out-of-Scope Use

    • Clinical Use: The model is NOT intended for clinical care, clinical decision-making, or use as a medical device/diagnostic tool.
    • Commercial Deployment: Any deployed use case, commercial or otherwise, is considered out of scope. The models are intended for research use only.

  3. Install and set up the GigaTIME environment

    main

    GigaTIME is recommended to be used with Conda and Python 3.11. For optimal reproducibility, A100 GPUs are used. Ensure the torch version in environment.yml matches your GPU and CUDA driver setup before proceeding.

    To create the environment:

    conda env create -f environment.yml

    To activate the environment:

    conda activate gigatime
    conda env create -f environment.yml
conda activate gigatime

  4. Load GigaTIME pre-trained models from HuggingFace

    main

    GigaTIME models require you to agree to the terms on HuggingFace. Once access is granted, set your HuggingFace read-only token as an environment variable to avoid connection errors:

    export HF_TOKEN=<huggingface read-only token>

    To load the model weights in Python:

    from huggingface_hub import snapshot_download
import torch
import os

repo_id = "prov-gigatime/GigaTIME"
local_dir = snapshot_download(repo_id=repo_id)

weights_path = os.path.join(local_dir, "model.pth")
state_dict = torch.load(weights_path, map_location="cpu")
model.load_state_dict(state_dict)

  5. GigaTIME-Flash: Efficient inference and whole-slide processing

    main

    GigaTIME-Flash is an optimized version built on top of GigaPath-Flash. It provides:

    • 6× faster inference
    • 8× less GPU memory usage
    • Better prediction quality compared to the original GigaTIME.

    Available Tutorials:

  7. Inference Resolution and Stitching Logic

    main

    A critical aspect of the GigaTIME inference pipeline is the distinction between inference resolution and visualization resolution.

    1. Inference (Native): Inference is performed on native level-0 512 x 512 tiles. The H&E image is never downsized before being fed to the model. This ensures maximum fidelity for the prediction.
    2. Stitching (Visualization): Because a slide-level map of native-resolution predictions would be too large to handle, the predictions are downsampled using the STITCH_BLOCK parameter. Each tile's 512 x 512 prediction is resized to a STITCH_BLOCK x STITCH_BLOCK area (e.g., 8x8) and placed into the global slide-level canvas.

    This approach preserves high-accuracy local predictions while allowing for manageable global visualization.

  8. Run inference with GigaTIME-flash

    main

    GigaTIME-flash is a lightweight model that converts H&E [B, 3, 256, 256] tiles into 23-channel virtual multiplex IF (mIF) maps [B, 23, 256, 256].

    To use it, you must:

    1. Set your HuggingFace read-only token as an environment variable.
    2. Download the model weights and configuration from HuggingFace.
    3. Initialize the GigaTIMEFlash model architecture.
    4. Remap the checkpoint keys to match the model's internal structure (handling module., .base_layer., and encoder. prefixes).
    5. Preprocess H&E tiles using preprocess_tile and run them through do_inference.
    export HF_TOKEN=<huggingface read-only token>
    from huggingface_hub import snapshot_download
import torch
import os

repo_id = "prov-gigatime/GigaTIME-flash"
local_dir = snapshot_download(repo_id=repo_id)
weights_path = os.path.join(local_dir, "model.pth")

# Initialize model architecture
model = GigaTIMEFlash(num_classes=23)

# Load and remap weights
checkpoint = torch.load(weights_path, map_location="cpu")
if isinstance(checkpoint, dict) and "state_dict" in checkpoint:
    checkpoint = checkpoint["state_dict"]

model_state = model.state_dict()
loaded = {}
for key, value in checkpoint.items():
    candidates = [key]
    if key.startswith("module."):
        candidates.append(key[len("module."):])
    if ".base_layer." in key:
        candidates.append(key.replace(".base_layer.", "."))
    if key.startswith("encoder.") and not key.startswith("encoder.base_model.model."):
        candidates.append(key.replace("encoder.", "encoder.base_model.model.", 1))
    for candidate in candidates:
        if candidate in model_state and model_state[candidate].shape == value.shape:
            loaded[candidate] = value
            break

model.load_state_dict(loaded, strict=False)
model.to(device).eval()

  9. Perform TCGA Slide-Level Inference with GigaTIME-Flash

    main

    This workflow enables whole-slide virtual multiplex immunofluorescence (mIF) inference on TCGA H&E slides. It processes the slide by tiling it into native-resolution 512 x 512 tiles, running inference on each, and stitching the results into a slide-level map across 23 channels.

    Workflow Steps:

    1. Load Slide: Open a TCGA .svs slide using openslide or tiffslide.
    2. Tissue Masking: Build a tissue mask at a low-resolution level to identify valid areas.
    3. Tiling: Tile the slide into native-resolution 512 x 512 tiles (Level 0) over the tissue.
    4. Inference: Run inference using either GigaTIME-Flash (faster, lower memory) or the original GigaTIME.
    5. Stitching: Stitch per-tile predictions into a slide-level canvas for visualization.

    Key Configuration Parameters:

    • MODEL_VARIANT: Set to "flash" (default) or "gigatime".
    • PATCH_SIZE: Set to 512 (native level-0 tile size; the model sees these without resizing).
    • STRIDE: Set to 512 for non-overlapping tiles.
    • STITCH_BLOCK: Controls the downsampling factor for the visualization canvas (e.g., 8 means each tile's prediction is reduced to an 8x8 block in the final map).
    • MAX_PATCHES: Set to an integer for a quick smoke test; None processes the entire slide.
    # Example configuration snippet
MODEL_VARIANT = "flash"          # "flash" (default) or "gigatime"
SLIDE_PATH = "path/to/your/slide.svs"
INFERENCE_LEVEL = 0              # read tissue at full resolution
PATCH_SIZE = 512                # native level-0 tile size
STRIDE = 512                    # non-overlapping native tiles
STITCH_BLOCK = 8                # visualization downsampling factor

  10. Implement the training and validation loops

    main

    The training loop follows these steps:

    1. Set model to model.train().
    2. For each batch, downsample the target by a factor of 8 and resize it to the input dimensions to account for pixel-level registration errors.
    3. Perform a forward pass, calculate loss using the specified criterion, and update weights via backpropagation.
    4. Calculate box-based metrics (Pearson/Spearman) for evaluation.

    Validation follows a similar pattern but uses model.eval() and torch.no_grad() to disable gradient computation and dropout/batch-norm updates.

  11. Filter training data based on image quality and segmentation metrics

    main

    When preparing the dataset, it is recommended to filter tile pairs to ensure high-quality training. The following criteria are used in the training pipeline:

    1. Image Quality:
      • Black ratio < 0.3
      • Variance > 200 (Applied to both COMET and H&E images)
    2. Segmentation Quality:
      • Dice coefficient > 0.2

  12. Install Slide Reader Dependencies

    main

    To read .svs or other whole-slide images, you must install a slide reader backend. The notebook supports openslide or tiffslide.

    • Option 1 (Recommended for system stability): Install openslide via your system package manager and then install the Python bindings.
      brew install openslide && pip install openslide-python
    • Option 2 (Pure Python): Use tiffslide, which has no system dependencies.
      pip install tiffslide
    # For macOS
brew install openslide && pip install openslide-python

# Or pure python
pip install tiffslide