MedSigLIP

MedSigLIP is a variant of SigLIP (Sigmoid Loss for Language Image Pre-training) specifically trained to encode medical images and text into a shared embedding space. It is designed for medical image interpretation tasks that do not require text generation.

Key Capabilities:

• Use Cases: Data-efficient classification, zero-shot classification, and semantic image retrieval.
• Model Architecture: Contains a 400M parameter vision encoder and a 400M parameter text encoder.
• Input Specifications: Supports 448x448 image resolution and up to 64 text tokens.

Note: If your application requires text generation, use MedGemma instead.

The MedSigLIP serving framework is a Python library designed to simplify the creation of custom prediction containers for Vertex AI. It provides a structured way to build HTTP servers that comply with Vertex AI custom container requirements.

Developers should use this framework to implement model-specific HTTP servers by wrapping custom data handling and orchestration logic within the provided framework structures.

The pre-processor configuration files required for MedSigLIP can be downloaded from the official Hugging Face repository. These files are necessary for correctly preparing input data for the model.

Download them from: https://huggingface.co/google/siglip-base-patch16-224/tree/main

To begin using MedSigLIP, you can follow these paths depending on your deployment preference:

1. Developer Documentation: Access the full range of next steps and the model card via the official developer portal.
2. Hugging Face / Model Garden: Access the model weights directly on Hugging Face or via Google Cloud's Model Garden.
3. Local/Notebook Exploration: Use the provided notebooks in this repository to learn how to use the model with Hugging Face and Vertex AI.
4. Vertex AI Deployment: Use the implementation found in python/serving to build a container for deployment to Vertex AI.

MedSigLIP supports two primary prediction methods in Vertex AI:

• Online predictions: Synchronous requests made to a deployed Vertex AI Endpoint. These are served with low latency and are ideal for production environments where immediate responses are required. Costs are based on the active time of the virtual machine hosting the endpoint.
• Batch predictions: Asynchronous requests run on a set of inputs (images or text) specified in a single job. These are made directly to a model in the Vertex AI Model Registry without requiring a deployed endpoint. They are ideal for generating embeddings for large datasets (e.g., for training) where low latency is not a priority. Costs are based on the time the virtual machine spends running the specific prediction job.

While the transformers image processor handles resizing automatically using PIL.Image.resize, you can use tf.image.resize to exactly match the resizing implementation used in MedSigLIP evaluations (based on the Big Vision library).

Use tf.image.resize with method='bilinear' and antialias=False to target the INPUT_DIMENSION (e.g., [448, 448]).

from tensorflow.image import resize as tf_resize
import numpy as np
from PIL import Image

INPUT_DIMENSION = [448, 448]

def resize(image):
    return Image.fromarray(
        tf_resize(
            images=image, size=INPUT_DIMENSION, method='bilinear', antialias=False
        ).numpy().astype(np.uint8)
    )

imgs = [resize(img) for img in imgs]