SAM 2 (Segment Anything Model 2)

The SA-V training set provides mp4 videos and masklet annotations stored as JSON files per video. Automatic and manual masklets are stored in separate files: {video_id}_auto.json and {video_id}_manual.json. These files can be loaded as Python dictionaries.

{
    "video_id"                        : str; video id
    "video_duration"                  : float64; the duration in seconds of this video
    "video_frame_count"               : float64; the number of frames in the video
    "video_height"                    : float64; the height of the video
    "video_width"                     : float64; the width of the video
    "video_resolution"                : float64; video_height $\times$ video_width
    "video_environment"               : List[str]; "Indoor" or "Outdoor"
    "video_split"                     : str; "train" for training set
    "masklet"                         : List[List[Dict]]; masklet annotations in list of list of RLEs. 
                                        The outer list is over frames in the video and the inner list 
                                        is over objects in the video.
    "masklet_id"                      : List[int]; the masklet ids
    "masklet_size_rel"                : List[float]; the average mask area normalized by resolution 
                                        across all the frames where the object is visible
    "masklet_size_abs"                : List[float]; the average mask area (in pixels) 
                                        across all the frames where the object is visible
    "masklet_size_bucket"             : List[str]; "small": $1$ <= masklet_size_abs < $32^2$,
                                        "medium": $32^2$ <= masklet_size_abs < $96^2$,
                                        and "large": masklet_size_abs > $96^2$
    "masklet_visibility_changes"      : List[int]; the number of times where the visibility changes 
                                        after the first appearance (e.g., invisible -> visible 
                                        or visible -> invisible)
    "masklet_first_appeared_frame"    : List[int]; the index of the frame where the object appears 
                                        the first time in the video. Always 0 for auto masklets.
    "masklet_frame_count"             : List[int]; the number of frames being annotated. Note that 
                                        videos are annotated at 6 fps (annotated every 4 frames) 
                                        while the videos are at 24 fps.
    "masklet_edited_frame_count"      : List[int]; the number of frames being edited by human annotators. 
                                        Always 0 for auto masklets.
    "masklet_type"                    : List[str]; "auto" or "manual"
    "masklet_stability_score"         : Optional[List[List[float]]]; per-mask stability scores. Auto annotation only.
    "masklet_num"                     : int; the number of manual/auto masklets in the video
}

The validation and test sets are provided as extracted JPEG frames (at 24 fps) and PNG masks (annotated at 6 fps). The directory structure is as follows:

• sav_val(sav_test).txt: List of video IDs.
• JPEGImages_24fps/: Contains subdirectories for each {video_id}, with individual .jpg frames.
• Annotations_6fps/: Contains subdirectories for each {video_id}, which then contain subdirectories for each object (e.g., 000, 001), containing the .png masks for that object in specific frames.

sav_val(sav_test)
├── sav_val.txt (sav_test.txt): a list of video ids in the split
├── JPEGImages_24fps # videos are extracted at 24 fps
│   ├── {video_id}
│   │     ├── 00000.jpg        # video frame
│   │     ├── 00001.jpg        # video frame
│   │     ├── 00002.jpg        # video frame
│   │     ├── 00003.jpg        # video frame
│   │     └── ...
│   ├── {video_id}
│   ├── {video_id}
│   └── ...
└── Annotations_6fps # videos are annotated at 6 fps
    ├── {video_id}
    │   ├── 000               # obj 000
    │   │    ├── 00000.png    # mask for object 000 in 00000.jpg
    │   │    ├── 00004.png    # mask for object 000 in 00004.jpg
    │   │    ├── 00008.png    # mask for object 000 in 00008.jpg
    │   │    ├── 00012.png    # mask for object 000 in 00012.jpg
    │   │    └── ...
    │   ├── 001               
    │   ├── 002               
    │   └── ...
    ├── {video_id}
    ├── {video_id}
    └── ...

This guide describes how to fine-tune SAM 2 checkpoints using the MOSE dataset.

Requirements:

• Hardware: A100 GPUs with 80 GB of memory are assumed.
• Dataset: Download the MOSE dataset from the MOSE API.

Steps:

1. Configure Paths: In configs/sam2.1_training/sam2.1_hiera_b+_MOSE_finetune.yaml, set the following keys to your local MOSE paths:
   • dataset.img_folder: Path to MOSE JPEGImages folder.
   • dataset.gt_folder: Path to MOSE Annotations folder.
   • dataset.file_list_txt: (Optional) Path to a filelist containing a subset of videos.
2. Run Training: Execute the training script using training/train.py.

Logging:

• Checkpoints and logs are saved to sam2_logs by default.
• You can customize the log directory in the config using experiment_log_dir.
• Monitor training losses via tensorboard logs located in the tensorboard/ subdirectory of your experiment log directory.

python training/train.py \
    -c configs/sam2.1_training/sam2.1_hiera_b+_MOSE_finetune.yaml \
    --use-cluster 0 \
    --num-gpus 8

You can bypass manual checkpoint downloading by loading models directly from Hugging Face using the .from_pretrained() method. This requires pip install huggingface_hub.

Image Prediction via Hugging Face

import torch
from sam2.sam2_image_predictor import SAM2ImagePredictor

predictor = SAM2ImagePredictor.from_pretrained("facebook/sam2-hiera-large")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    predictor.set_image(<your_image>)
    masks, _, _ = predictor.predict(<input_prompts>)

Video Prediction via Hugging Face

import torch
from sam2.sam2_video_predictor import SAM2VideoPredictor

predictor = SAM2VideoPredictor.from_pretrained("facebook/sam2-hiera-large")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state(<your_video>)
    frame_idx, object_ids, masks = predictor.add_new_points_or_box(state, <your_prompts>)

    for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
        ...

SAM 2 requires python>=3.10, torch>=2.5.1, and torchvision>=0.20.1.

Standard Installation

To install SAM 2 on a GPU machine:

git clone https://github.com/facebookresearch/sam2.git && cd sam2

pip install -e .

Installation with Notebook Support

To use the SAM 2 predictor and run the provided example notebooks, install the [notebooks] extra:

pip install -e ".[notebooks]"

Important Notes

• Windows Users: It is strongly recommended to use Windows Subsystem for Linux (WSL) with Ubuntu.
• CUDA Kernels: Installation requires compiling a custom CUDA kernel using nvcc. Ensure your CUDA toolkit version matches your PyTorch CUDA version.
• CUDA Extension Errors: If you see Failed to build the SAM 2 CUDA extension, you can ignore it; the model will still work, though some post-processing functionality may be limited.

To utilize MPS (Metal Performance Shaders) for hardware acceleration on macOS, you must run the backend service outside of Docker.

1. Environment Setup

First, create and activate a Conda environment, install ffmpeg, and install the demo dependencies from the SAM 2 root directory:

conda create --name sam2-demo python=3.10 --yes
conda activate sam2-demo
conda install -c conda-forge ffmpeg
pip install -e '.[interactive-demo]'

2. Download Checkpoints

Ensure you have the model checkpoints downloaded:

(cd ./checkpoints && ./download_ckpts.sh)

3. Launch the Server

Navigate to demo/backend/server/ and run the server using gunicorn. You must provide several environment variables to configure the model and paths.

cd demo/backend/server/

PYTORCH_ENABLE_MPS_FALLBACK=1 \
APP_ROOT="$(pwd)/../../../" \
API_URL=http://localhost:7263 \
MODEL_SIZE=base_plus \
DATA_PATH="$(pwd)/../../data" \
DEFAULT_VIDEO_PATH=gallery/05_default_juggle.mp4 \
gunicorn \
    --worker-class gthread app:app \
    --workers 1 \
    --threads 2 \
    --bind 0.0.0.0:7263 \
    --timeout 60