DOCSEARCHbyAlgolia
Submit Docs

InSPyReNet Documentation

repository·main·Indexed 19 days ago

https://github.com/plemeri/inspyrenet

A PyTorch implementation of an image pyramid-based framework for High Resolution Salient Object Detection (SOD). InSPyReNet (Inverse Saliency Pyramid Reconstruction Network) leverages an image pyramid structure and blending techniques to achieve high-resolution saliency map prediction. It supports training via single or multi-GPU (DDP), inference on images, videos, and webcams, and is also available as a simplified PyPI package called `transparent-background` for background removal.

Tokens
5.6K
Snippets
19
Records
23
Agent score
73%

What's inside InSPyReNet

  1. Overview of InSPyReNet

    main
    InSPyReNet (Inverse Saliency Pyramid Reconstruction Network) is an image pyramid-based framework for High Resolution (HR) Salient Object Detection (SOD). It is designed to produce a strict image pyramid structure of saliency maps, allowing for the ensembling of multiple results via pyramid-based image blending. This approach enables high-resolution prediction without requiring high-resolution training datasets by synthesizing pyramids from pairs of low-resolution (LR) and high-resolution (HR) scales.

  2. Use pre-trained InSPyReNet checkpoints

    main

    To use the pre-trained checkpoints provided in the Model Zoo, you must ensure that the latest.pth file is placed in the directory specified by the checkpoint_dir setting in your configuration file.

    For example, if you are using the InSPyReNet_SwinB.yaml configuration, you must locate the checkpoint_dir path within that file and place the downloaded latest.pth there.

    Checkpoints are categorized by their training datasets and scales (Low Resolution vs. High Resolution).

    # Example: Check your config file for the checkpoint directory
# In configs/InSPyReNet_SwinB.yaml:
# checkpoint_dir: /path/to/your/checkpoints/

# Then place the downloaded latest.pth in that directory:
# /path/to/your/checkpoints/latest.pth

  3. Download datasets and model checkpoints using easy download

    main

    You can download all necessary data, including ImageNet pre-trained backbone checkpoints, training/testing datasets, pre-trained model checkpoints, and pre-computed saliency maps, using the provided utility script.

    Use the --extra flag to include results from supplementary materials and DIS5K. Use the --dest flag to specify a custom destination directory (the script will automatically create symbolic links in data and snapshots folders).

    python utils/download.py --extra --dest [DEST]

  4. Prepare datasets and backbone checkpoints

    main

    InSPyReNet requires specific directory structures and files for training and testing.

    Dataset Structure

    All datasets must follow this structure:

    • [Dataset.root]/images/: Contains input images.
    • [Dataset.root]/masks/: Contains ground truth masks.

    Configuration Paths

    Refer to your .yaml config file for the following keys:

    • Train.Dataset.root: The base directory for training data.
    • Test.Dataset.root: The base directory for testing data.
    • Train.Dataset.sets: A list of dataset folder names to include in training (e.g., ['DUTS-TR', 'HRSOD-TR']).

    Required Files

    • Backbone Checkpoints: Must be placed in data/backbone_ckpt/*.pth (e.g., Res2Net50 or SwinB).
    • Training Datasets: Located in data/Train_Dataset/....
    • Testing Datasets: Located in data/Test_Dataset/....

  5. Test and Evaluate benchmarks

    main

    To run inference on test benchmarks and evaluate the resulting metrics, use run/Test.py and run/Eval.py respectively.

    Test Benchmarks

    python run/Test.py --config configs/InSPyReNet_SwinB.yaml --verbose

    Evaluate Metrics

    python run/Eval.py --config configs/InSPyReNet_SwinB.yaml --verbose

  6. Run the All-in-One (Train, Test, Eval) command

    main

    The Expr.py script allows you to run the entire pipeline (Training, Testing, and Evaluation) in a single command.

    All-in-One Commands

    Single GPU:

    python Expr.py --config configs/InSPyReNet_SwinB.yaml --verbose

    Multi-GPU (DDP) with 4 GPUs:

    torchrun --standalone --nproc_per_node=4 Expr.py --config configs/InSPyReNet_SwinB.yaml --verbose

    Multi-GPU (DDP) with designated devices (e.g., 2 GPUs - 0 and 1):

    CUDA_VISIBLE_DEVICES=0,1 torchrun --standalone --nproc_per_node=2 Expr.py --config configs/InSPyReNet_SwinB.yaml --verbose
    # Single GPU
python Expr.py --config configs/InSPyReNet_SwinB.yaml --verbose

# Multi GPUs with DDP (e.g., 4 GPUs)
torchrun --standalone --nproc_per_node=4 Expr.py --config configs/InSPyReNet_SwinB.yaml --verbose

# Multi GPUs with DDP with designated devices (e.g., 2 GPUs - 0 and 1)
CUDA_VISIBLE_DEVICES=0,1 torchrun --standalone --nproc_per_node=2 Expr.py --config configs/InSPyReNet_SwinB.yaml --verbose

  7. Train InSPyReNet

    main

    You can train the model using a single GPU or multiple GPUs via Distributed Data Parallel (DDP).

    Training Commands

    Single GPU:

    python run/Train.py --config configs/InSPyReNet_SwinB.yaml --verbose

    Multi-GPU (DDP) with 4 GPUs:

    torchrun --standalone --nproc_per_node=4 run/Train.py --config configs/InSPyReNet_SwinB.yaml --verbose

    Multi-GPU (DDP) with specific devices (e.g., GPUs 0 and 1):

    CUDA_VISIBLE_DEVICES=0,1 torchrun --standalone --nproc_per_node=2 run/Train.py --config configs/InSPyReNet_SwinB.yaml --verbose

    Training CLI Arguments

    • --config, -c [CONFIG_FILE]: Path to the training configuration YAML file.
    • --resume, -r: Resume training from the last saved checkpoint.
    • --verbose, -v: Output progress information to the console.
    • --debug, -d: Save debug images at every epoch.
    # Single GPU
python run/Train.py --config configs/InSPyReNet_SwinB.yaml --verbose

# Multi GPUs with DDP (e.g., 4 GPUs)
torchrun --standalone --nproc_per_node=4 run/Train.py --config configs/InSPyReNet_SwinB.yaml --verbose

# Multi GPUs with DDP with designated devices (e.g., 2 GPUs - 0 and 1)
CUDA_VISIBLE_DEVICES=0,1 torchrun --standalone --nproc_per_node=2 run/Train.py --config configs/InSPyReNet_SwinB.yaml --verbose

  8. Use InSPyReNet via the transparent-background package

    main

    For a simplified experience, InSPyReNet is available as a PyPI package called transparent-background. This package provides both a Command-line Tool and a Python API for background removal and salient object detection. For detailed usage instructions, refer to the transparent-background documentation.

    pip install transparent-background

  9. Install InSPyReNet and set up the environment

    main

    To use InSPyReNet, create a Conda environment, install PyTorch, and then install the project requirements.

    1. Create and activate the environment:
      conda create -y -n inspyrenet python
conda activate inspyrenet
    2. Install PyTorch (example for Linux with CUDA 11.6):
      pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116
    3. Install project dependencies:
      pip install -r requirements.txt
    conda create -y -n inspyrenet python
conda activate inspyrenet
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116
pip install -r requirements.txt

  10. Run Inference on images, videos, or webcams

    main

    Use run/Inference.py to perform salient object detection on your own data. Supported formats include .jpg, .jpeg, .png for images, and .mp4, .mov, .avi for videos. Webcam input is supported via device index (e.g., 0).

    Inference Command

    python run/Inference.py --config configs/InSPyReNet_SwinB.yaml --source [SOURCE] --dest [DEST] --type [TYPE] --gpu --jit --verbose

    Arguments

    • --source [SOURCE]: The input data. Can be a single file, a folder of files, or a webcam index (e.g., 0).
    • --dest [DEST] (optional): Destination folder for results. Defaults to the results folder.
    • --type [TYPE]: The output mode:
      • map: Saliency map only.
      • green: Replaces background with a green screen.
      • rgba: Generates an RGBA output (alpha map based on saliency score). Note: Not supported for video/webcam.
      • blur: Blurs the background.
      • overlay: Covers the salient object with translucent green and highlights edges.
      • [FILE]: Use another image file (e.g., background.png) as the new background.
    • --gpu: Enables GPU acceleration.
    • --jit: Enables Just-In-Time compilation to improve inference speed.
    • --verbose: Displays progress information.
    python run/Inference.py --config configs/InSPyReNet_SwinB.yaml --source image.png --dest ./output --type map --gpu --jit --verbose

  11. Access pre-computed saliency maps for SOD benchmarks

    main

    InSPyReNet provides pre-computed saliency maps for various Salient Object Detection (SOD) benchmarks. You can select a model configuration based on your needs (e.g., InSPyReNet_Res2Net50 or InSPyReNet_SwinB) and download the corresponding results for specific datasets like DUTS-TE, DUT-OMRON, ECSSD, HKU-IS, PASCAL-S, or DAVIS-S via the provided links in the model zoo documentation.

    | Config | DUTS-TE | DUT-OMRON | ECSSD | HKU-IS | PASCAL-S | DAVIS-S | HRSOD-TE | UHRSD-TE |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| [InSPyReNet_Res2Net50](../configs/InSPyReNet_Res2Net50.yaml) | [Link](...) | [Link](...) | [Link](...) | [Link](...) | [Link](...) | N/A | N/A | N/A |
| [InSPyReNet_SwinB](../configs/InSPyReNet_SwinB.yaml) | [Link](...) | [Link](...) | [Link](...) | [Link](...) | [Link](...) | [Link](...) | [Link](...) | [Link](...) |

  12. Access pre-computed saliency maps for DIS5K results

    main

    For the Dichotomous Image Segmentation (DIS5K) dataset, pre-computed saliency maps are available for two main configurations:

    1. InSPyReNet_SwinB_DIS5K_LR: Optimized for Low Resolution (LR).
    2. InSPyReNet_SwinB_DIS5K: Standard configuration.

    Results can be downloaded for various splits including DIS-VD, DIS-TE1, DIS-TE2, DIS-TE3, and DIS-TE4.

    | Config | DIS-VD | DIS-TE1 | DIS-TE2 | DIS-TE3 | DIS-TE4 |
| :--- | :--- | :--- | :--- | :--- | :--- |
| [InSPyReNet_SwinB_DIS5K_LR](../configs/extra_dataset/InSPyReNet_SwinB_DIS5K_LR.yaml) | [Link](...) | [Link](...) | [Link](...) | [Link](...) | [Link](...) |
| [InSPyReNet_SwinB_DIS5K](../configs/extra_dataset/InSPyReNet_SwinB_DIS5K.yaml) | [Link](...) | [Link](...) | [Link](...) | [Link](...) | [Link](...) |