DOCSEARCHbyAlgolia
Submit Docs

CellSAM Foundation Model

repository·master·Indexed 41 days ago

https://github.com/vanvalenlab/cellsam

A foundation model for state-of-the-art cell segmentation across various cellular targets (bacteria, yeast, tissue) and imaging modalities (brightfield, fluorescence, phase). It provides a Python library featuring the `cellsam_pipeline` for automated inference, a `segment_cellular_image` function, and an interactive Napari plugin for manual annotation and segmentation.

Tokens
4.3K
Snippets
22
Records
26
Agent score
53%

What's inside CellSAM

  1. Evaluate Anchor DETR on 8 GPUs

    master

    To perform evaluation on a single node with 8 GPUs, use the --eval flag along with --coco_path and --resume (to specify the path to your checkpoint file).

    python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --eval --coco_path /path/to/coco --resume /path/to/checkpoint.pth

  2. Predict all cell masks using cellsam_pipeline

    master

    For most biological use cases, use cellsam_pipeline to automatically detect and segment all cells in an image. When no bounding boxes are provided, the CellFinder module automatically identifies bounding boxes for all cells. The pipeline automatically handles weight downloading and will use a GPU if available, falling back to CPU otherwise.

    Key parameters for cellsam_pipeline:

    • img: The input image.
    • use_wsi: Boolean (default behavior for whole slide images).
    • low_contrast_enhancement: Boolean to enable/disable contrast enhancement.
    • gauge_cell_size: Boolean to enable/disable cell size gauging.
    from cellSAM import cellsam_pipeline
import imageio.v3 as iio

# Load image
img = iio.imread("path_to_image.png")

# Run inference pipeline to find all cells
mask = cellsam_pipeline(
    img, 
    use_wsi=False, 
    low_contrast_enhancement=False, 
    gauge_cell_size=False
)

  3. Set up a reproducibility environment for CellSAM

    master

    To reproduce the results from the publication, it is recommended to use a clean virtual environment. You can use any environment manager (e.g., uv, pixi, conda, or venv).

    1. Create and activate a new environment:
    python3.XX -m venv cs-eval-env
source cs-eval-env/bin/activate
    1. Install the cellSAM library from the parent directory:
    pip install ..
    1. Install the specific requirements for the evaluation suite:
    pip install -r requirements.txt

    Note: Installing evaluation requirements may downgrade certain dependencies like torch or numpy from the base cellSAM installation.

    python3.XX -m venv cs-eval-env
source cs-eval-env/bin/activate
pip install ..
pip install -r requirements.txt

  5. Download the CellSAM training dataset

    master

    The CellSAM training dataset is approximately 14GB. Ensure you have sufficient disk space and network bandwidth before proceeding.

    You can download the compressed dataset for local use using the download_training_data() function. The files will be stored in $HOME/.deepcell/data/cellsam/.

    from cellSAM import download_training_data

download_training_data()

  6. Install CellSAM via pip

    master

    The easiest way to install CellSAM is directly from the GitHub repository using pip. Ensure you have python>=3.10 installed, as the package relies on PyTorch.

    pip install git+https://github.com/vanvalenlab/cellSAM.git

  7. Perform interactive cell segmentation in Napari

    master

    The Napari plugin allows for manual, interactive segmentation by defining bounding boxes around individual cells.

    Workflow:

    1. Load an image: Use the File menu -> Open File(s) to load your image (e.g., .png).
    2. Select Layer: In the left panel, select the "Bounding Boxes" layer.
    3. Select Tool: In the upper-left tool panel, select the "Add Rectangle" option.
    4. Draw Box: Select a region corresponding to a single cell. Note: The plugin assumes there is exactly one cell per bounding box.
    5. Confirm: Press the "Confirm Annot." button on the right panel. The model will then predict the segmentation mask for that cell.

    The resulting masks are displayed in the "Segmentation Overlay" layer.

    Scaling Note: For segmenting many cells or entire images, use the cellSAM.cellsam_pipeline.cellsam_pipeline library interface instead of the interactive plugin.

    # No code block provided for the interactive UI workflow, but the steps are:
# 1. cellsam napari
# 2. Select 'Bounding Boxes' layer
# 3. Use 'Add Rectangle' tool
# 4. Click 'Confirm Annot.'

  8. Clear segmentation results in Napari

    master

    You can manage or remove segmentation masks using the following controls in the Napari interface:

    • Remove individual predictions: Use the standard napari labeling tools within the Segmentation Overlay layer.
    • Clear all predictions: Click the Clear mask button on the right panel to remove all segmentation masks from the current session.
    • Reset session: Click the Reset button on the right panel to remove both the current predictions and the underlying image. Use this to start fresh with a new image.

  9. Segment specific cells using prompting with get_model

    master

    For manual labeling or specific use cases, you can use get_model to access the model directly and provide bounding box prompts.

    Bounding Box Format Requirements: When passing prompts to predictor.predict, the bounding boxes must follow the format (x1, x2, y1, y2), where (x1, y1) is the lower-left corner and (x2, y2) is the upper-right corner.

    Workflow:

    1. Get the model via get_model().
    2. Preprocess the image using format_image_shape, normalize_image, and appropriate numpy transpositions/expansions.
    3. Call predictor.predict(im, boxes_per_heatmap=[[[x1, y1, x2, y2]]]).

    Note: The boxes_per_heatmap argument expects a nested list structure containing the coordinates.

    import numpy as np
from cellSAM import get_model
from cellSAM.utils import format_image_shape, normalize_image

# 1. Get model
predictor = get_model()

# 2. Prepare image (example coordinates: x1, y1, w, h)
x1, y1, w, h = 290, 365, 60, 38
x2, y2 = x1 + w, y1 + h

im = format_image_shape(img)
im = normalize_image(im)
im = np.transpose(im, (2, 0, 1))
im = np.expand_dims(im, axis=0)

# 3. Predict with bounding box prompt (x1, y1, x2, y2)
pred_mask, *_ = predictor.predict(im, boxes_per_heatmap=[[[x1, y1, x2, y2]]])

  10. Download CellSAM model weights

    master

    Pre-trained model weights are required for the inference pipeline. You can acquire them in two ways:

    1. Automatic: The weights are automatically downloaded to $HOME/.deepcell/models the first time you execute the cellSAM.cellsam_pipeline.cellsam_pipeline function.
    2. Manual: Call get_model() without any arguments to trigger a download of the latest pre-trained weights.
    # Example of manual download
from cellSAM import get_model
get_model()

  11. Segment an entire image using the Napari plugin

    master

    The plugin includes a Segment All button on the right panel for automatic segmentation of all cells in an image.

    ⚠️ Caution: This button is intended for interactive use and is not recommended if:

    • The image dimensions are greater than 2k x 2k pixels.
    • You do not have a local GPU available.

    For large-scale or high-throughput segmentation, use the cellSAM.cellsam_pipeline.cellsam_pipeline library interface.

  12. Download and prepare the evaluation dataset

    master

    The evaluation dataset can be downloaded using download_training_data().

    • Download location: The compressed archive is saved to $HOME/.deepcell/datasets/cellsam-data_v{X.Y}.tar.gz.
    • Size: The compressed archive is ~14GB and expands to ~84GB.

    To unpack the dataset to a specific location (e.g., /data) using multiple threads for faster decompression, you can use tar with pigz:

    tar --use-compress-program="unpigz -p 8" -xf $HOME/.deepcell/datasets/cellsam-data_v1.2.tar.gz -C /data

    After unpacking, the data will be available at /data/cellsam_v1.2.

    from cellSAM import download_training_data

download_training_data()