Grounded SAM 2

Grounding DINO is an open-set object detection model that marries DINO with grounded pre-training. It allows for detecting objects using natural language prompts (an image, text pair).

Key features:

• Open-Set Detection: Detect objects using language descriptions.
• High Performance: Achieves 52.5 AP on COCO zero-shot and 63.0 AP on COCO fine-tune.
• Flexibility: Can be integrated with Stable Diffusion or GLIGEN for controllable image editing.

Grounded SAM 2 is a foundation model pipeline designed to ground and track objects in videos. It integrates several open-world models including Grounding DINO, Grounding DINO 1.5 & 1.6, Florence-2, DINO-X, and SAM 2.

Key capabilities include:

• Ground and Segment Anything: Using various DINO models and SAM 2.
• Ground and Track Anything: Using various DINO models and SAM 2.
• Visualization: Leveraging the supervision library for detection, segmentation, and tracking visualization.

If the CUDA extension fails to build, the installation will proceed but post-processing will be skipped. To force the build and ensure errors are raised if it fails, use the SAM2_BUILD_ALLOW_ERRORS=0 environment variable.

Note: You must uninstall the existing version and remove any existing .so files before retrying.

pip uninstall -y SAM-2 && \
rm -f ./sam2/*.so && \
SAM2_BUILD_ALLOW_ERRORS=0 pip install -v -e ".[notebooks]"

For large-scale training across multiple nodes using SLURM, use the --use-cluster 1 flag and specify the number of nodes and cluster parameters.

python training/train.py \
    -c configs/sam2.1_training/sam2.1_hiera_b+_MOSE_finetune.yaml \
    --use-cluster 1 \
    --num-gpus 8 \
    --num-nodes 2 \
    --partition $PARTITION \
    --qos $QOS \
    --account $ACCOUNT

Before installation, you must download the pretrained checkpoints for both SAM 2 and Grounding DINO using the provided scripts.

# Download SAM 2 checkpoints
cd checkpoints
bash download_ckpts.sh

# Download Grounding DINO checkpoints
cd gdino_checkpoints
bash download_ckpts.sh

For advanced tracking scenarios:

1. Continuous ID (Experimental): Attempts to find new objects and assign new IDs across the whole video. Supports only the mask prompt type.

   • python grounded_sam2_tracking_demo_with_continuous_id.py
   • python grounded_sam2_tracking_demo_with_continuous_id_gd1.5.py (with API token)
   • Customizable params: text, video_dir, output_dir, output_video_path, step, box_threshold, text_threshold.

2. Continuous ID + Reverse Tracking: Covers the whole lifetime of the object.

   • python grounded_sam2_tracking_demo_with_continuous_id_plus.py

3. Real-Time Camera Tracking: Enables real-time tracking with ID continuity from a live camera stream.

   • python grounded_sam2_tracking_camera_with_continuous_id.py

Install SAM 2 on a GPU machine by cloning the repository and installing the package in editable mode. To run example notebooks, you must also install the [demo] extra which includes jupyter and matplotlib.

git clone https://github.com/facebookresearch/segment-anything-2.git

cd segment-anything-2; pip install -e .

# To include demo dependencies (jupyter, matplotlib)
pip install -e ".[demo]"

To install without Docker, ensure you have a Python 3.10 environment with torch >= 2.3.1, torchvision >= 0.18.1, and cuda-12.1.

Note on CUDA: Grounding DINO requires a CUDA compilation environment to compile the Deformable Attention operator. You may need to manually set the CUDA_HOME environment variable.

Follow these steps:

1. Install PyTorch and TorchVision with CUDA support.
2. Set CUDA_HOME if necessary.
3. Install the local package and Grounding DINO.

Combine SAM 2 tracking with Grounding DINO for open-set object segmentation and tracking in videos.

Standard Tracking: python grounded_sam2_tracking_demo.py (Saves results to ./tracking_results and video to children_tracking_demo_video.mp4).

Tracking with Grounding DINO 1.5/1.6: python grounded_sam2_tracking_demo_with_gd1.5.py (Requires API token).

Supported Prompt Types for Tracking:

• Point Prompt: Uniformly sampled points from the prediction mask.
• Box Prompt: Direct use of Grounding DINO box outputs.
• Mask Prompt: SAM 2 mask prediction results based on Grounding DINO box outputs.

To install Grounded SAM 2, ensure your environment meets the requirements and then install the package from the repository root.

Requirements

• OS: Linux (for Windows users, use WSL with Ubuntu).
• Python: $\ge$ 3.10 (recommended).
• PyTorch: $\ge$ 2.3.1 and a matching torchvision version.
• CUDA Toolkit: Must match your PyTorch CUDA version (typically CUDA 12.1).

Standard Installation

Install the package in editable mode with notebook support:

pip install -e ".[notebooks]"

Skip SAM 2 CUDA Extension

If you want to skip building the CUDA extension (which skips the post-processing step of removing small holes/sprinkles in masks), use the SAM2_BUILD_CUDA=0 environment variable:

SAM2_BUILD_CUDA=0 pip install -e ".[notebooks]"

To install Grounding DINO, ensure your CUDA_HOME environment variable is set to your CUDA toolkit path (e.g., /usr/local/cuda). If CUDA_HOME is not set, the package will compile in CPU-only mode. Failure to follow installation steps strictly may result in a NameError: name '_C' is not defined.

1. Clone the repository:

   git clone https://github.com/IDEA-Research/GroundingDINO.git
   cd GroundingDINO/

2. Install dependencies:

   pip install -e .

3. Download pre-trained weights:

   mkdir weights && cd weights
   wget -q https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth
   cd ..