DOCSEARCHbyAlgolia
Submit Docs

YOLACT (You Only Look At CoefficienTs)

repository·master·Indexed 26 days ago

https://github.com/dbolya/yolact

A fully convolutional model for real-time instance segmentation, including YOLACT and YOLACT++ versions. The repository provides tools for training on COCO and Pascal SBD datasets, quantitative and qualitative evaluation, and multi-GPU training. It includes an implementation of Deformable Convolutional Networks V2 (DCNv2) and Deformable ROI Pooling (DCNPooling) for PyTorch 1.0 and 0.4.

Tokens
2.7K
Snippets
8
Records
14
Agent score
90%

What's inside YOLACT

  1. Install YOLACT++ Deformable Convolutional Layers

    master

    To use YOLACT++, you must compile the DCNv2 (deformable convolutional layers) code. Ensure you have the latest CUDA toolkit installed from NVIDIA's website.

    cd external/DCNv2
python setup.py build develop

  2. Configure a custom dataset in data/config.py

    master

    To use a custom dataset, you must define it in data/config.py by copying the dataset_base object and then updating the yolact_base_config to point to your new dataset definition.

    1. Define the dataset

    Create a new dictionary object using dataset_base.copy().

    Requirements:

    • Annotation Format: Use COCO-style Object Detection JSON. You may omit info, license, and certain image fields (license, flickr_url, coco_url, date_captured).
    • Class IDs: In your JSON, class IDs must start at 1 and increase sequentially according to the order in class_names. If your IDs do not follow this (e.g., standard COCO format), use the label_map field in dataset_base to map them.
    • Validation: If you don't provide a separate validation split, train.py will default to evaluating the first 5000 images every 2 epochs.

    2. Register the dataset

    In yolact_base_config, update the 'dataset' key to match the name of your new configuration object.

    # In data/config.py

my_custom_dataset = dataset_base.copy({
    'name': 'My Dataset',

    'train_images': 'path_to_training_images',
    'train_info':   'path_to_training_annotation',

    'valid_images': 'path_to_validation_images',
    'valid_info':   'path_to_validation_annotation',

    'has_gt': True,
    'class_names': ('my_class_id_1', 'my_class_id_2', 'my_class_id_3', ...)
})

# Then update the base config:
yolact_base_config['dataset'] = 'my_custom_dataset'

  3. Evaluate YOLACT models (Quantitative and Qualitative)

    master

    Use eval.py to perform various evaluation tasks. Place your weight files in the ./weights directory first. The config name is the prefix of the weight file (e.g., yolact_base for yolact_base_54_800000.pth).

    Common Tasks:

    • Quantitative Validation: Evaluate on the entire validation set.
    • Output COCO JSON: Generate JSON files for detection and instance segmentation (useful for submission or run_coco_eval.py).
    • Qualitative Display: Display results visually on the COCO dataset.
    • Benchmarking: Run the raw model on a subset of images (e.g., first 1000).
    • Image Processing: Process single images, specific files, or entire folders.
    • Video Processing: Process video files or real-time webcam/video feeds.

  5. Train on Pascal SBD dataset

    master

    To train YOLACT on the Pascal SBD dataset, follow these steps:

    1. Download the benchmark.tgz dataset from the official source.
    2. Extract the dataset and move the dataset/img folder to ./data/sbd/img (relative to the YOLACT root).
    3. Download the COCO-style annotations from this Google Drive link.
    4. Extract these annotations into ./data/sbd/.
    5. Run training using the specific config: --config=yolact_resnet50_pascal_config.

    Note: You can verify results using the pre-trained yolact_resnet50_pascal_config weights available here.

  6. Train YOLACT models

    master

    To train YOLACT, you must first download an ImageNet-pretrained model (e.g., for Resnet101, Resnet50, or Darknet53) and place it in the ./weights directory.

    Training Options:

    • --config: Specify the configuration (e.g., yolact_base_config).
    • --batch_size: Set the batch size. For 550px models, 1 batch uses ~1.5GB VRAM.
    • --resume: Resume training from a specific weight file. Use --start_iter=-1 to start from the iteration specified in the weight file.
    • --no_log: Disable default training and validation logging.

    Note: You can interrupt training with Ctrl+C; it will save an *_interrupt.pth file at the current iteration.

  7. Build Deformable Convolutional Networks V2

    master

    To build the DCNv2 extension and run the included examples and gradient checks, use the following commands in the terminal:

    1. Run the build script: ./make.sh
    2. Run the test script: python test.py
    ./make.sh         # build
python test.py    # run examples and gradient check

  8. Download COCO Datasets for Training or Evaluation

    master

    Use the provided scripts to download the necessary COCO data.

    • For Training: Download the COCO dataset and 2014/2017 annotations. This will download approximately 21GB of files into ./data/coco.
    • For test-dev Evaluation: Download the test-dev dataset using the specific test script.
    sh data/scripts/COCO.sh
# or for test-dev
sh data/scripts/COCO_test.sh

  9. Configure Multi-GPU Training

    master

    YOLACT supports multi-GPU training.

    1. Set the visible GPUs using the CUDA_VISIBLE_DEVICES environment variable (e.g., export CUDA_VISIBLE_DEVICES=0,1,2,3).
    2. Set the --batch_size to 8 * num_gpus. The script automatically scales hyperparameters.
    3. (Optional) Use --batch_alloc=[alloc] to specify a comma-separated list of images per GPU. This list must sum to the total batch_size.

  10. Install YOLACT

    master

    To install YOLACT, clone the repository and set up the environment using either Anaconda or pip.

    Using Anaconda: Run conda env create -f environment.yml.

    Using pip:

    1. Set up a Python 3 environment.
    2. Install PyTorch 1.0.1 (or higher) and TorchVision.
    3. Install required packages. Note: cython must be installed before pycocotools.
    pip install cython
pip install opencv-python pillow pycocotools matplotlib
    git clone https://github.com/dbolya/yolact.git
cd yolact

  11. Run the YOLACT demo web server

    master

    The YOLACT demo includes a simple Python-based web server used to host the demonstration interface. It serves detection indices and images from the COCO dataset.

    Warning: This server uses SimpleHTTPRequestHandler and contains unsafe path handling practices (e.g., direct path translation) that are not suitable for production environments. It is intended for local demonstration purposes only.

    To run the server, execute the web/server.py script. By default, it listens on port 6337.

  12. Use DCNPooling (Deformable ROI Pooling) in PyTorch

    master

    The DCNPooling class implements deformable ROI pooling (V2), wrapping offset and mask calculations. It requires an input tensor and a tensor of ROIs.

    ROI Format: ROIs should be a tensor of shape (N, 5) where the columns are [batch_index, x, y, x + w, y + h].

    Usage Example:

    from dcn_v2 import DCNPooling
import torch

input = torch.randn(2, 32, 64, 64).cuda()
# batch_inds, x, y, w, h
batch_inds = torch.randint(2, (20, 1)).cuda().float()
x = torch.randint(256, (20, 1)).cuda().float()
y = torch.randint(256, (20, 1)).cuda().float()
w = torch.randint(64, (20, 1)).cuda().float()
h = torch.randint(64, (20, 1)).cuda().float()

# Concatenate to form ROIs: [batch_idx, x1, y1, x2, y2]
rois = torch.cat((batch_inds, x, y, x + w, y + h), dim=1)

# Initialize DCNPooling
dpooling = DCNPooling(spatial_scale=1.0 / 4,
                     pooled_size=7,
                     output_dim=32,
                     no_trans=False,
                     group_size=1,
                     trans_std=0.1).cuda()

dout = dpooling(input, rois)
    from dcn_v2 import DCNPooling
input = torch.randn(2, 32, 64, 64).cuda()
batch_inds = torch.randint(2, (20, 1)).cuda().float()
x = torch.randint(256, (20, 1)).cuda().float()
y = torch.randint(256, (20, 1)).cuda().float()
w = torch.randint(64, (20, 1)).cuda().float()
h = torch.randint(64, (20, 1)).cuda().float()
rois = torch.cat((batch_inds, x, y, x + w, y + h), dim=1)

# mdformable pooling (V2)
# wrap all things (offset and mask) in DCNPooling
dpooling = DCNPooling(spatial_scale=1.0 / 4,
                     pooled_size=7,
                     output_dim=32,
                     no_trans=False,
                     group_size=1,
                     trans_std=0.1).cuda()

dout = dpooling(input, rois)