DEIMv2 Object Detection Framework

DEIMv2 is an evolution of the DEIM framework that leverages features from DINOv3 for real-time object detection. It offers a range of model sizes (Atto, Femto, Pico, N, S, M, L, and X) designed to be adaptable for various scenarios, from ultra-lightweight applications to high-performance requirements. The S-sized model achieves over 50 AP on the COCO benchmark.

Key features include:

• Support for multiple vision tasks (object detection, instance segmentation, and human pose estimation) via related projects like EdgeCrafter.
• Integration with Hugging Face for model access.
• Optimized attention modules for memory efficiency in S and M models.

Backbone requirements vary by model variant:

• HGNetv2-based variants: Backbones are downloaded automatically during training.
• DEIMv2-L and X: Uses DINOv3-S and S+. Download them following the DINOv3 guide.
• DEIMv2-S and M: Uses distilled ViT-Tiny and ViT-Tiny+. Download them from the provided Google Drive links and ViT-Tiny+.

Place all downloaded checkpoints in the ./ckpts folder:

ckpts/
├── dinov3_vits16.pth
├── vitt_distill.pt
├── vittplus_distill.pt
└── ...

To train on a custom dataset, follow these steps:

1. Disable category remapping: In your configuration, set remap_mscoco_category: False to prevent automatic remapping to MSCOCO IDs.
2. Organize directory structure:

   dataset/
   ├── images/
   │   ├── train/
   │   └── val/
   └── annotations/
       ├── instances_train.json
       └── instances_val.json

3. Ensure COCO format: Annotations must be in COCO JSON format.
4. Update configuration: Modify configs/dataset/custom_detection.yml with your num_classes, img_folder, and ann_file paths.

To set up the environment, create a new Conda environment with Python 3.11 and install the required dependencies via pip. It is recommended to use PyTorch 2.5.1, 2.4.1, or any version 2.0 or higher.

# Recommended PyTorch versions: 2.5.1 or 2.4.1 (2.0+ required)
conda create -n deimv2 python=3.11 -y
conda activate deimv2
pip install -r requirements.txt

Use the FiftyOne integration to visualize DEIMv2 model outputs and detections.

# Setup
pip install fiftyone

# Run FiftyOne Visualization
python tools/visualization/fiftyone_vis.py -c configs/deimv2/deimv2_dinov3_${model}_coco.yml -r model.pth

You can benchmark the model for architectural metrics (FLOPs, MACs, Params) or measure TensorRT latency using the provided benchmark tools.

# Setup
pip install -r tools/benchmark/requirements.txt

# Get Model FLOPs, MACs, and Params
python tools/benchmark/get_info.py -c configs/deimv2/deimv2_dinov3_${model}_coco.yml

# Measure TensorRT Latency
python tools/benchmark/trt_benchmark.py --COCO_dir path/to/COCO2017 --engine_dir model.engine

To train with a specific input size (e.g., 320x320):

1. Set eval_spatial_size: [320, 320].
2. Update train_dataloader.dataset.transforms.ops to include a Resize operation with the target size. For Mosaic augmentation, it is recommended that output_size is half the input_size (e.g., 160 for a 320 input).
3. Update collate_fn.base_size to match the input size.

eval_spatial_size: [320, 320]

train_dataloader:
  total_batch_size: 64 
  dataset: 
    transforms:
      ops:
        - {type: Mosaic, output_size: 160, ...}
        - {type: Resize, size: [320, 320] }
  collate_fn:
    base_size: 320

val_dataloader:
  dataset:
    transforms:
      ops:
        - {type: Resize, size: [320, 320] }

To use the COCO2017 dataset:

1. Download COCO2017 from OpenDataLab or COCO.
2. Update the paths in configs/dataset/coco_detection.yml to point to your local img_folder and ann_file for both train_dataloader and val_dataloader.

train_dataloader:
    img_folder: /data/COCO2017/train2017/
    ann_file: /data/COCO2017/annotations/instances_train2017.json
val_dataloader:
    img_folder: /data/COCO2017/val2017/
    ann_file: /data/COCO2017/annotations/instances_val2017.json