DOCSEARCHbyAlgolia
Submit Docs

Segment Everything Everywhere All at Once (SEEM)

repository·v1.0·Indexed 26 days ago

https://github.com/ux-decoder/segment-everything-everywhere-all-at-once

SEEM is a multi-modal segmentation model built on the X-Decoder architecture. It supports a wide range of prompts, including visual (points, boxes, scribbles, images) and language (text, audio) inputs, to generate object masks and semantic labels. The project includes version 1.0, with SEEM_v0 supporting single object interaction and SEEM_v1 supporting multiple objects. It provides capabilities for click/scribble-to-mask, text-to-mask, referring image-to-mask, and audio-to-mask segmentation.

Tokens
7K
Snippets
13
Records
23
Agent score
88%

What's inside SEEM

  1. Overview of SEEM capabilities

    v1.0

    SEEM (Segment Everything Everywhere All at Once) is a multi-modal segmentation model that supports various prompt types for interactive segmentation:

    • Versatility: Supports clicks, boxes, polygons, scribbles, text, and referring images.
    • Compositionality: Handles any combination of the prompt types mentioned above.
    • Interactivity: Supports multi-round interaction using a memory prompt to store session history.
    • Semantic Awareness: Can assign semantic labels to predicted masks.

    Version Differences:

    • SEEM_v0: Supports single interactive object training and inference.
    • SEEM_v1: Supports multiple interactive objects training and inference.

  2. Supported Segmentation Modalities in SEEM

    v1.0

    SEEM (Segment Everything Everywhere All at Once) supports multiple multi-modal interaction types to generate object masks and semantic category labels:

    • Click/Scribble to Mask: Generate masks and category labels using simple user clicks or strokes.
    • Text to Mask: Perform grounding segmentation by providing text prompts.
    • Referring Image to Mask: Segment objects in a target image that share similar semantics with an object identified in a referring image (supports spatial relationship awareness).
    • Referring Image to Video Mask: Segment objects in videos using queries specified via a referring image (requires no video-specific training).
    • Audio to Mask: Uses Whisper to convert audio input into text prompts for segmentation.

  3. Train SEEM with SAM Backbone (ViT-B and ViT-L)

    v1.0

    To train SEEM using a Segment Anything Model (SAM) backbone, use the configs/seem/samvitb_unicl_lang_v1.yaml (for ViT-B) or configs/seem/samvitl_unicl_lang_v1.yaml (for ViT-L) configuration files. When using SAM, you must provide the pretrained SAM weights via the MODEL.BACKBONE.PRETRAINED override.

    # Example for SAM ViT-B
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 mpirun -n 8 python entry.py train \
            --conf_files configs/seem/samvitb_unicl_lang_v1.yaml \
            --overrides \
            FP16 True \
            COCO.INPUT.IMAGE_SIZE 1024 \
            MODEL.DECODER.HIDDEN_DIM 512 \
            MODEL.ENCODER.CONVS_DIM 512 \
            MODEL.ENCODER.MASK_DIM 512 \
            TEST.BATCH_SIZE_TOTAL 8 \
            TRAIN.BATCH_SIZE_TOTAL 16 \
            TRAIN.BATCH_SIZE_PER_GPU 2 \
            SOLVER.MAX_NUM_EPOCHS 50 \
            SOLVER.BASE_LR 0.0001 \
            SOLVER.FIX_PARAM.backbone True \
            SOLVER.FIX_PARAM.lang_encoder True \
            SOLVER.FIX_PARAM.pixel_decoder True \
            MODEL.DECODER.COST_SPATIAL.CLASS_WEIGHT 5.0 \
            MODEL.DECODER.COST_SPATIAL.MASK_WEIGHT 2.0 \
            MODEL.DECODER.COST_SPATIAL.DICE_WEIGHT 2.0 \
            MODEL.DECODER.TOP_SPATIAL_LAYERS 10 \
            MODEL.DECODER.SPATIAL.ENABLED True \
            MODEL.DECODER.GROUNDING.ENABLED True \
            FIND_UNUSED_PARAMETERS True \
            ATTENTION_ARCH.SPATIAL_MEMORIES 32 \
            MODEL.DECODER.SPATIAL.MAX_ITER 5 \
            ATTENTION_ARCH.QUERY_NUMBER 3 \
            STROKE_SAMPLER.MAX_CANDIDATE 10 \
            MODEL.BACKBONE.PRETRAINED /pth/to/xdecoder_data/pretrained/sam_vit_b_01ec64.pth \
            WEIGHT True \
            RESUME_FROM /pth/to/xdecoder_data/pretrained/focalb_lang_unicl.pt

  4. Use the SEEM interactive demo

    v1.0

    To use the web-based interactive demo:

    1. Try the default examples provided.
    2. Upload an image you wish to segment.
    3. Select one or more prompt types (e.g., clicks, boxes, scribbles, text, or referring images).
    4. Important: Provide the actual prompt data for each selected type (e.g., if you select 'referring image', you must upload a second image and draw on it).
    5. Note on vocabulary: The model defaults to the COCO 80 categories. For open-vocabulary segmentation, use the 'text' button to provide a label after drawing scribbles.
    6. Click "Submit" and wait for the results.

  5. Install SEEM and X-Decoder dependencies

    v1.0

    To set up the environment, ensure you are on a Linux system with a CUDA-enabled GPU. For evaluation, you need >8GB VRAM; for training, >12GB VRAM is required. Install the necessary Python packages using the provided requirements files. If you are training the deformable vision encoder, you must also compile the custom operators. If you are running the SEEM demo, install ffmpeg via apt.

    # Python Package Installation
pip install -r assets/requirements/requirements.txt
pip install -r assets/requirements/requirements_custom.txt

# Customer Operator [only need training deformable vision encoder]
cd modeling/vision/encoder/ops && sh make.sh && cd ../../../../

# System Package [only need for demo in SEEM]
sudo apt update
sudo apt install ffmpeg

  6. Prepare 4M Image Text Pairs for X-Decoder

    v1.0

    X-Decoder requires 4M image-text pairs stored as .arrow files under .xdecoder_data/pretrain_arrows_code224. This follows the ViLT data preparation.

    For a quick start/debugging, you can download a single hacked test set file. However, for full model training, you must include the full list of .arrow files (including code224_vg.arrow, code224_sbu_*.arrow, and code224_conceptual_caption_train_*.arrow).

    # Download coco karparthy test set (we hack the training data to be coco_caption_karpathy_test.arrow only for quick start in the codebase)
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/coco_caption_karpathy_test.arrow

  7. Prepare Evaluation Datasets

    v1.0

    Depending on your model (SEEM or X-Decoder), follow these instructions for evaluation datasets:

    • RefCOCO (SEEM & X-Decoder): Follow the COCO preparation steps.
    • ADE20K & Cityscapes (X-Decoder): Follow Mask2Former instructions.
    • BDD100K (X-Decoder): Download the 10k split from BDD100K website.
    • PascalVOC & Interactive Datasets (SEEM): Follow RITM instructions. The expected structure is:
    .xdecoder_data/PascalVOC/
├── Annotations/
├── ImageSets/
├── JPEGImages/
├── SegmentationClass/
└── SegmentationObject/

  8. Prepare evaluation data and environment variables

    v1.0

    Download and unzip coco_caption.zip to your .xdecoder_data directory for evaluation. You must then configure several environment variables to point to your data directory so the system can locate datasets and Java runtimes.

    # save coco_caption.zip to .xdecoder_data
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/coco_caption.zip
unzip coco_caption.zip

# Environment Variables
export DETECTRON2_DATASETS=/pth/to/xdecoder_data
export DATASET=/pth/to/xdecoder_data
export DATASET2=/pth/to/xdecoder_data
export VLDATASET=/pth/to/xdecoder_data
export PATH=$PATH:/pth/to/xdecoder_data/coco_caption/jre1.8.0_321/bin
export PYTHONPATH=$PYTHONPATH:/pth/to/xdecoder_data/coco_caption

  9. Prepare COCO datasets for SEEM & X-Decoder

    v1.0

    To prepare the COCO training datasets for SEEM and X-Decoder, ensure your base COCO datasets (panoptic_train2017, panoptic_semseg_train2017, etc.) are structured exactly like Mask2Former. Then, download the additional logistic and custom annotation files into the .xdecoder_data/coco/annotations directory.

    For SEEM specifically, you must also download LVIS annotations for mask preparation into the .xdecoder_data/lvis directory.

    # (SEEM & X-Decoder) Download additional logistic and custom annotation files to .xdecoder_data/coco/annotations
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/caption_class_similarity.pth
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/captions_train2017_filtrefgumdval_filtvlp.json
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/grounding_train2017_filtrefgumdval_filtvlp.json
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/panoptic_train2017_filtrefgumdval_filtvlp.json
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/refcocog_umd_val.json
wget https://github.com/peteanderson80/coco-caption/blob/master/annotations/captions_val2014.json

# (SEEM) Download LVIS annotations for mask preparation
wget https://huggingface.co/xdecoder/SEEM/resolve/main/coco_train2017_filtrefgumdval_lvis.json

  10. Train SEEM (Focal-T and Focal-L)

    v1.0

    To train SEEM models, use python entry.py train with the corresponding configuration file. Use configs/seem/focalt_unicl_lang_v1.yaml for Focal-T and configs/seem/focall_unicl_lang_v1.yaml for Focal-L. Key parameters include SOLVER.MAX_NUM_EPOCHS, SOLVER.BASE_LR, and RESUME_FROM for pretrained weights.

    # Example for Focal-T
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 mpirun -n 8 python entry.py train \
            --conf_files configs/seem/focalt_unicl_lang_v1.yaml \
            --overrides \
            FP16 True \
            COCO.INPUT.IMAGE_SIZE 1024 \
            MODEL.DECODER.HIDDEN_DIM 512 \
            MODEL.ENCODER.CONVS_DIM 512 \
            MODEL.ENCODER.MASK_DIM 512 \
            TEST.BATCH_SIZE_TOTAL 8 \
            TRAIN.BATCH_SIZE_TOTAL 16 \
            TRAIN.BATCH_SIZE_PER_GPU 2 \
            SOLVER.MAX_NUM_EPOCHS 50 \
            SOLVER.BASE_LR 0.0001 \
            SOLVER.FIX_PARAM.backbone True \
            SOLVER.FIX_PARAM.lang_encoder True \
            SOLVER.FIX_PARAM.pixel_decoder True \
            MODEL.DECODER.COST_SPATIAL.CLASS_WEIGHT 5.0 \
            MODEL.DECODER.COST_SPATIAL.MASK_WEIGHT 2.0 \
            MODEL.DECODER.COST_SPATIAL.DICE_WEIGHT 2.0 \
            MODEL.DECODER.TOP_SPATIAL_LAYERS 10 \
            MODEL.DECODER.SPATIAL.ENABLED True \
            MODEL.DECODER.GROUNDING.ENABLED True \
            FIND_UNUSED_PARAMETERS True \
            ATTENTION_ARCH.SPATIAL_MEMORIES 32 \
            MODEL.DECODER.SPATIAL.MAX_ITER 5 \
            ATTENTION_ARCH.QUERY_NUMBER 3 \
            STROKE_SAMPLER.MAX_CANDIDATE 10 \
            WEIGHT True \
            RESUME_FROM /pth/to/xdecoder_data/pretrained/xdecoder_focalt_last.pt

  11. Evaluate SEEM models

    v1.0

    To evaluate SEEM models, use the python entry.py evaluate command with mpirun. Configuration is handled via --conf_files and runtime parameters are passed through --overrides.

    Supported model variants include:

    • Focal-T (v0 and v1)
    • ViT-B SAM v1
    • ViT-L SAM v1
    • Focal-L v0
    # Example for SEEM Focal-T v0
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 mpirun -n 8 python entry.py evaluate \
            --conf_files configs/seem/focalt_unicl_lang_v0.yaml \
            --overrides \
            COCO.INPUT.IMAGE_SIZE 1024 \
            MODEL.DECODER.HIDDEN_DIM 512 \
            MODEL.ENCODER.CONVS_DIM 512 \
            MODEL.ENCODER.MASK_DIM 512 \
            VOC.TEST.BATCH_SIZE_TOTAL 8 \
            TEST.BATCH_SIZE_TOTAL 8 \
            REF.TEST.BATCH_SIZE_TOTAL 8 \
            FP16 True \
            WEIGHT True \
            RESUME_FROM /pth/to/xdecoder_data/seem/seem_focalt_v0.pt