DOCSEARCHbyAlgolia
Submit Docs

SAM 3 (Segment Anything Model 3)

repository·main·Indexed 27 days ago

https://github.com/facebookresearch/sam3

A unified foundation model for promptable segmentation in images and videos. SAM 3 supports open-vocabulary segmentation using text prompts, points, boxes, and masks. The repository includes implementations for image and video predictors, tools for running inference evaluations on SA-Co datasets (Gold and Silver), and Jupyter notebook examples.

Tokens
10.9K
Snippets
34
Records
57
Agent score
95%

What's inside SAM 3

  1. Prepare Ego4D frame dataset

    main

    To prepare Ego4D frames:

    1. Accept the license agreement on the official Ego4D website.
    2. Configure AWS credentials using awscli with the values provided in your Ego4D email.
    3. Install the ego4d library.
    4. Update CONFIG_FRAMES.yaml with path_annotations, aws_access_key_id, aws_secret_access_key, and ego4d_path (where frames are saved).
    5. (Optional) Set remove_downloaded_videos_ego4d in CONFIG_FRAMES.yaml to control video cleanup.
    6. Download the clips subset and extract frames.
    pip install awscli
aws configure

# Install library
pip install ego4d

# Download and extract
python download_videos.py ego4d
python extract_frames.py ego4d

  2. Prepare Berkeley Driving Dataset (BDD) 100k image dataset

    main

    To prepare BDD100k images, download the 100K Images dataset from the BDD100k website, extract it, and run the preprocessing script.

    Note: You must provide the path to the Silver annotations folder containing silver_bdd100k_merged_test.json.

    python preprocess_silver_geode_bdd100k_food_rec.py --annotation_file <FOLDER_WITH_SILVER_ANNOTATIONS>/silver_bdd100k_merged_test.json --raw_images_folder <RAW_BDD_IMAGES_FOLDER> --processed_images_folder <PROCESSED_BDD_IMAGES_FOLDER> --dataset_name bdd100k

  4. Prepare National Gallery of Art (NGA) image dataset

    main

    To prepare NGA images, run the download and preprocess script. This script handles both downloading raw images and preprocessing them for evaluation.

    Note: You must provide the path to the Silver annotations folder containing silver_nga_art_merged_test.json.

    python download_preprocess_nga.py --annotation_file <FOLDER_WITH_SILVER_ANNOTATIONS>/silver_nga_art_merged_test.json --raw_images_folder <RAW_NGA_IMAGES_FOLDER> --processed_images_folder <PROCESSED_NGA_IMAGES_FOLDER>

  5. Monitor training with TensorBoard

    main

    Training outputs are saved to the experiment_log_dir specified in your config. This directory contains:

    • config.yaml and config_resolved.yaml (configurations)
    • checkpoints/ (model weights)
    • tensorboard/ (logs for visualization)
    • logs/ (text logs)
    • submitit_logs/ (cluster job logs)

    To monitor progress, run: tensorboard --logdir /path/to/experiment_log_dir/tensorboard.

  7. Request access to SAM 3 checkpoints

    main
    Before using SAM 3, you must request access to the checkpoints on the SAM 3 Hugging Face repository. Once access is granted, you must authenticate your environment (e.g., using hf auth login) to download the checkpoints.

  8. Evaluate SAM3 on SA-Co/Silver benchmark

    main

    To reproduce the SAM3 evaluation on the SA-Co/Silver benchmark, follow these steps:

    1. Configure Paths: Edit sam3/train/configs/eval_base.yaml to include the local paths where you downloaded the images and annotations.
    2. Run Inference: Use the sam3/train/train.py script with a specific subset configuration. There are 10 subsets available.

    Local Execution (adjust --num-gpus as needed):

    python sam3/train/train.py -c configs/silver_image_evals/sam3_gold_image_bdd100k.yaml --use-cluster 0 --num-gpus 1

    SLURM Cluster Execution: First, edit eval_base.yaml to reflect your SLURM configuration (e.g., partition, qos). Then run:

    python sam3/train/train.py -c configs/silver_image_evals/sam3_gold_image_bdd100k.yaml --use-cluster 1

    Predictions are saved to the directory specified in eval_base.yaml.

    python sam3/train/train.py -c configs/silver_image_evals/sam3_gold_image_bdd100k.yaml --use-cluster 0 --num-gpus 1

  9. Run SAM3 inference evaluation

    main

    To reproduce the SAM3 evaluation, follow these steps:

    1. Configure paths: Edit sam3/train/configs/eval_base.yaml with the local paths to your downloaded images and annotations.
    2. Run locally: Use the sam3/train/train.py script with the appropriate config for the subset. Use --use-cluster 0 for local execution and specify the number of GPUs with --num-gpus.
    3. Run on SLURM: Use --use-cluster 1 after editing eval_base.yaml with your cluster's partition and QOS settings.

    Predictions will be saved to the folder specified in eval_base.yaml.

  10. Install SAM 3

    main

    To install SAM 3, ensure you meet the prerequisites and follow the installation steps using Conda and pip.

    Prerequisites

    • Python 3.12 or higher
    • PyTorch 2.7 or higher
    • CUDA-compatible GPU with CUDA 12.6 or higher

    Installation Steps

    1. Create and activate a new Conda environment.
    2. Install PyTorch with CUDA support.
    3. Clone the repository and install the package in editable mode.
    4. Install optional dependencies for notebooks or development.
    5. (Optional) Install dependencies for faster inference.
    # 1. Create a new Conda environment
conda create -n sam3 python=3.12
conda deactivate
conda activate sam3

# 2. Install PyTorch with CUDA support
pip install torch==2.10.0 torchvision --index-url https://download.pytorch.org/whl/cu128

# 3. Clone the repository and install the package
git clone https://github.com/facebookresearch/sam3.git
cd sam3
pip install -e .

# 4. Install additional dependencies for example notebooks or development
pip install -e ".[notebooks]"
pip install -e ".[train,dev]"

# 5. Optional dependencies for faster inference
pip install einops ninja && pip install flash-attn-3 --no-deps --index-url https://download.pytorch.org/whl/cu128
pip install git+https://github.com/ronghanghu/cc_torch.git

  12. Prepare Food Recognition Challenge 2022 image dataset

    main

    To prepare Food Recognition images, download the [Round 2] public_validation_set_2.0.tar.gz file from the challenge website, extract it, and run the preprocessing script.

    Note: You must provide the path to the Silver annotations folder containing silver_food_rec_merged_test.json.

    python preprocess_silver_geode_bdd100k_food_rec.py --annotation_file <FOLDER_WITH_SILVER_ANNOTATIONS>/silver_food_rec_merged_test.json --raw_images_folder <RAW_FOOD_IMAGES_FOLDER> --processed_images_folder <PROCESSED_FOOD_IMAGES_FOLDER> --dataset_name food_rec