RT-DETRv4 Documentation

Customizing Batch Size

To increase the total batch size (e.g., doubling it):

1. Update total_batch_size in configs/base/dataloader.yml.
2. Adjust learning rates and EMA settings in your model config (e.g., configs/rtv4/rtv4_hgnetv2_l_coco.yml) using linear scaling laws.

Customizing Input Size

To train with a specific input size (e.g., 320x320):

1. Update transforms in configs/base/dataloader.yml for both train_dataloader and val_dataloader.
2. Update eval_spatial_size in your model base config (e.g., base/rtv4_base.yml).

# Example: Batch Size adjustment in dataloader.yml
train_dataloader:
    total_batch_size: 64

# Example: Input Size adjustment in dataloader.yml
train_dataloader:
  dataset:
      transforms:
          ops:
              - {type: Resize, size: [320, 320], }
  collate_fn:
      base_size: 320

You can benchmark RT-DETRv4 models for architectural metrics (FLOPs, MACs, Params) or runtime latency using TensorRT.

1. Install dependencies: Install requirements from tools/benchmark/requirements.txt.
2. Get Model Info: Use tools/benchmark/get_info.py to calculate FLOPs, MACs, and Parameters.
3. TensorRT Latency: Use tools/benchmark/trt_benchmark.py to measure latency on a specific dataset (e.g., COCO) using a TensorRT engine.

# Setup
pip install -r tools/benchmark/requirements.txt

# Model FLOPs, MACs, and Params
python tools/benchmark/get_info.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml

# TensorRT Latency
python tools/benchmark/trt_benchmark.py --COCO_dir path/to/COCO2017 --engine_dir model.engine

Use the Voxel51 Fiftyone tool to visualize model predictions.

1. Install: pip install fiftyone.
2. Run Visualization: Execute tools/visualization/fiftyone_vis.py with your model configuration and weights.

pip install fiftyone
python tools/visualization/fiftyone_vis.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml -r model.pth

RT-DETRv4 supports inference on both images and videos using three different backends. Ensure you install the requirements for the specific tool first.

• ONNX Runtime: Use tools/inference/onnx_inf.py.
• TensorRT: Use tools/inference/trt_inf.py.
• PyTorch: Use tools/inference/torch_inf.py (supports specifying a CUDA device).

# Setup for ONNX/TRT/Torch inference
pip install -r tools/inference/requirements.txt

# Inference examples
python tools/inference/onnx_inf.py --onnx model.onnx --input image.jpg
python tools/inference/trt_inf.py --trt model.engine --input image.jpg
python tools/inference/torch_inf.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml -r model.pth --input image.jpg --device cuda:0

To set up the RT-DETRv4 environment, create a new Conda environment with Python 3.11.9 and install the required dependencies using pip.

conda create -n rtv4 python=3.11.9
conda activate rtv4
pip install -r requirements.txt

To use the COCO2017 dataset, download it from OpenDataLab or COCO, then update the paths in configs/dataset/coco_detection.yml to point to your local img_folder and ann_file for both train_dataloader and val_dataloader.

train_dataloader:
    img_folder: /data/COCO2017/train2017/
    ann_file: /data/COCO2017/annotations/instances_train2017.json
val_dataloader:
    img_folder: /data/COCO2017/val2017/
    ann_file: /data/COCO2017/annotations/instances_val2017.json

To deploy RT-DETRv4 models, you can export them to ONNX format and subsequently convert them to a TensorRT engine for optimized inference.

1. Install dependencies: Install onnx and onnxsim.
2. Export to ONNX: Use tools/deployment/export_onnx.py providing a configuration file and the model weights.
3. Export to TensorRT: Use the trtexec command to convert the .onnx file to a .engine file with FP16 precision.

# 1. Setup
pip install onnx onnxsim

# 2. Export onnx
python tools/deployment/export_onnx.py --check -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml -r model.pth

# 3. Export tensorrt
trtexec --onnx="model.onnx" --saveEngine="model.engine" --fp16

The repository provides utility scripts for common maintenance tasks:

• Auto Resume Training: Use the safe_training.sh script to handle training resumes.
• Converting Model Weights: Use tools/reference/convert_weight.py to transform existing .pth weights.

# Auto Resume Training
bash tools/reference/safe_training.sh

# Converting Model Weights
python tools/reference/convert_weight.py model.pth

RT-DETRv4 uses a Vision Foundation Model (VFM) as a teacher. To use the DINOv3 ViT-B/16-LVD-1689M model, download the weights and then update the teacher_model section in your model configuration file (e.g., ./configs/rtv4/rtv4_hgnetv2_${model}_coco.yml).

Set dinov3_repo_path to your local DINOv3 repository and dinov3_weights_path to the downloaded .pth file.

teacher_model:
  type: "DINOv3TeacherModel"
  dinov3_repo_path: dinov3/
  dinov3_weights_path: pretrain/dinov3_vitb16_pretrain_lvd1689m.pth