CenterTrack

CenterTrack is a simultaneous detection and tracking algorithm. Unlike traditional tracking-by-detection pipelines that separate these steps, CenterTrack takes the current frame, the previous frame, and a heatmap rendered from previous tracking results as input.

It predicts:

1. The current detection heatmap.
2. The offsets of detected objects to their centers in the previous frame.

This allows the model to localize objects and predict their temporal associations in a single pass. It is designed to be online (no future information used) and real-time. The method can be extended to monocular 3D tracking by regressing additional 3D attributes.

Pre-trained models for various tasks (MOT17, KITTI, nuScenes, COCO) can be downloaded from the official Google Drive repository.

Environment Requirements:

• PyTorch 1.0
• CUDA 10.0
• CUDNN 7.5

Model Categories:

• 2D Bounding Box Tracking: Models for MOT17 and KITTI 2D datasets.
• Monocular 3D Detection/Tracking: Models for the nuScenes dataset.
• COCO Tracking: Models provided for demonstration purposes only.

Google Drive Link

To evaluate 3D tracking performance on the nuScenes dataset, use the following command.

Key arguments:

• --exp_id: Experiment identifier.
• --load_model: Path to the pretrained .pth model.
• --dataset: Set to nuscenes.
• --track_thresh: Score threshold for predicting a bounding box.
• --pre_hm: Enables the input heatmap.

python test.py tracking,ddd --exp_id nuScenes_3Dtracking --load_model ../models/nuScenes_3Dtracking.pth --dataset nuscenes --track_thresh 0.1 --pre_hm

COCO is used for training demo systems for 80-category tracking or pose tracking.

1. Download the 2017 Train, Val, and Test images from the COCO website.
2. Download the 2017 train/val and test image info annotation files.
3. Organize the data in ${CenterTrack_ROOT}/data/coco as follows:

${CenterTrack_ROOT}
|-- data
`-- |-- coco
    |-- annotations
    |   |-- instances_train2017.json
    |   |-- instances_val2017.json
    |   |-- person_keypoints_train2017.json
    |   |-- person_keypoints_val2017.json
    |   |-- image_info_test-dev2017.json
    |---|-- train2017
    |---|-- val2017
    `---|-- test2017

nuScenes is used for 3D object tracking and pretraining KITTI models.

1. Download the "Keyframe blobs" (images only) and the maps/metadata from the nuScenes website.
2. Unzip, rename, and merge folders into ${CenterTrack_ROOT}/data/nuscenes/v1.0-trainval.
3. Run the conversion script in the tools directory to create COCO annotations:

python convert_kitti_to_coco.py

This will generate train.json, val.json, and test.json under data/nuscenes/annotations. Note that the nuscenes API is required for this preprocessing step.

To train on a custom dataset, use the --dataset custom flag. You must provide annotations in COCO format (refer to convert_X_to_coco.py in the tools directory for examples). You will also need to specify the annotation path, image path, input resolution, and number of classes.

Example command for training on a custom dataset (modeled after the MOT17 experiment):

python main.py tracking --exp_id mot17_half_sc --dataset custom --custom_dataset_ann_path ../data/mot17/annotations/train_half.json --custom_dataset_img_path ../data/mot17/train/ --input_h 544 --input_w 960 --num_classes 1 --pre_hm --ltrb_amodal --same_aug --hm_disturb 0.05 --lost_disturb 0.4 --fp_disturb 0.1 --gpus 0,1

KITTI Tracking is used for 2D tracking training and evaluation.

1. Download the images, annotations, and calibration information from the KITTI Tracking website and unzip them.
2. Place or symlink the data into ${CenterTrack_ROOT}/data/kitti_tracking following the required structure.
3. Run the conversion script in the tools directory to convert annotations to COCO format:

python convert_kitti_to_coco.py

The resulting structure should include a label_02_val_half and label_02_train_half directory, along with a annotations folder containing tracking_train.json, tracking_test.json, tracking_train_half.json, and tracking_val_half.json.

Training scripts are located in the experiments/ directory.

Guidelines for training:

• Experiment Names: The experiment names in the scripts correspond to the model names available in the model zoo.
• Resuming Training: If training is interrupted, append the --resume flag to your command. The system will automatically find the latest model associated with the same --exp_id.
• Pretraining: Some experiments require a model pretrained on a different dataset. You must either download the required pretrained model from the model zoo or train that model first before starting the current experiment.

To evaluate 2D vehicle tracking performance on the KITTI dataset, use the following command structure. Ensure the model is located in the models/ directory.

Key arguments:

• --exp_id: Experiment identifier.
• --dataset: Set to kitti_tracking.
• --dataset_version: e.g., val_half.
• --pre_hm: Enables the input heatmap.
• --track_thresh: Score threshold for predicting a bounding box.
• --load_model: Path to the pretrained .pth model.

python test.py tracking --exp_id kitti_half --dataset kitti_tracking --dataset_version val_half --pre_hm --track_thresh 0.4 --load_model ../models/kitti_half.pth