AB3DMOT Documentation

AB3DMOT is a real-time 3D multi-object tracking (MOT) baseline system designed for high performance and low computational cost.

Core Workflow:

1. Detection: Uses an off-the-shelf 3D object detector to obtain oriented 3D bounding boxes from LiDAR point clouds.
2. State Estimation: Employs a 3D Kalman filter.
3. Data Association: Uses the Hungarian algorithm to associate detections with existing tracks.

Key Features:

• High Speed: Capable of running at rates such as 214.7 FPS.
• 3D Focus: Provides 3D tracking results that can be projected to the 2D image plane for comparison with 2D MOT methods.
• New Metrics: Introduces new 3D MOT evaluation metrics (including sAMOTA) to extend the official KITTI 2D MOT evaluation.

To reproduce 2D MOT results for the KITTI test set, run the post-processing script:

python3 scripts/post_processing/trk_conf_threshold.py --dataset KITTI --result_sha pointrcnn_test_H1

After running, compress the following folder and upload it to the official KITTI evaluation server: ./results/KITTI/pointrcnn_test_H1_thres/data_0

To use the KITTI dataset with AB3DMOT, download the official KITTI multi object tracking dataset. You must include left color images, velodyne point clouds, GPS/IMU data, training labels, and camera calibration data.

Uncompress the data and organize it into the ./data/KITTI directory using the following structure:

AB3DMOT
├── data
│   ├── KITTI
│   │   ├── tracking
│   │   │   ├── training
│   │   │   │   ├──calib & velodyne & label_02 & image_02 & oxts
│   │   │   │   └── testing
│   │   │   │       ├──calib & velodyne & image_02 & oxts
│   │   │   └── detection
├── AB3DMOT_libs
└── configs

AB3DMOT requires the Xinshuo_PyToolbox repository. You must clone it and install its specific dependencies separately.

1. Clone the repository:

git clone https://github.com/xinshuoweng/Xinshuo_PyToolbox

2. Install its dependencies:

cd Xinshuo_PyToolbox
pip3 install -r requirements.txt
cd ..

If you have your own nuScenes detection results in the official JSON format, you must convert them to the KITTI-compatible format used by AB3DMOT.

Given a raw detection file at ./data/nuScenes/data/produced/results/detection/detname/results_val.json, run:

python3 scripts/nuScenes/export_kitti.py nuscenes_obj_result2kitti --result_name detname --split val

This creates files at ./data/nuScenes/nuKITTI/object/produced/results/val/detname/data.

Next, pre-process these files into the required input directory using:

python3 scripts/pre_processing/convert_det2input.py --dataset nuScenes --split val --det_name detname

The processed files will be located in ./data/nuScenes/detection/detname_cat_val (where cat is the category name, e.g., detname_Car_val).

To evaluate tracking results against the official nuScenes metrics on the validation set, convert the AB3DMOT results back to the nuScenes format and run the local evaluation script:

# 1. Convert KITTI-style results to nuScenes format
python3 scripts/nuScenes/export_kitti.py kitti_trk_result2nuscenes --result_name megvii_val_H1 --split val

# 2. Run evaluation
python3 scripts/nuScenes/evaluate.py --result_path ./results/nuScenes/megvii_val_H1/results_val.json

To run the tracker on the KITTI MOT validation set using the provided PointRCNN detections, execute the following command:

python3 main.py --dataset KITTI --det_name pointrcnn

This command generates:

• A folder pointrcnn_val_H1 containing combined results from all categories.
• Folders named pointrcnn_category_val_H1 for results per category.
• ./data_0 subfolders within result folders for MOT evaluation.
• ./trk_withid_0 subfolders for visualization (includes an ID in the last column).

To reproduce quantitative 3D MOT results on the KITTI MOT validation set, use the scripts/KITTI/evaluate.py script. You can specify different 3D IoU thresholds (e.g., 0.25, 0.5, or 0.7).

Example for 0.25 3D IoU threshold:

python3 scripts/KITTI/evaluate.py pointrcnn_val_H1 1 3D 0.25

Example for 0.5 3D IoU threshold:

python3 scripts/KITTI/evaluate.py pointrcnn_val_H1 1 3D 0.5

Example for 0.7 3D IoU threshold:

python3 scripts/KITTI/evaluate.py pointrcnn_Car_val_H1 1 3D 0.7

To install AB3DMOT, clone the repository and install the required dependencies. You can use either your system Python or a Python virtual environment. It is recommended to use a virtual environment to avoid dependency conflicts.

Using System Python

Navigate to the root of the cloned AB3DMOT directory and run:

cd path/to/AB3DMOT
pip3 install -r requirements.txt

Using a Virtual Environment (Recommended)

Create and activate a virtual environment, then install the requirements:

pip3 install venv
python3 -m venv env
source env/bin/activate
pip3 install -r requirements.txt

To run experiments on the nuScenes dataset, you must install specific versions of the following packages:

• nuscenes-devkit==1.1.9
• motmetrics<=1.1.3
• pandas>=0.24

To install these within your project's virtual environment, run:

cd path/to/AB3DMOT
source env/bin/activate
cd scripts/nuScenes
pip3 install -r requirements.txt
cd ../../

To reproduce quantitative 2D MOT results on the KITTI MOT validation set, run:

python3 scripts/KITTI/evaluate.py pointrcnn_val_H1 1 2D 0.5

AB3DMOT processes data in the KITTI format. To use nuScenes, follow these steps:

1. Download and Structure: Download the official nuScenes full dataset (v1.0), uncompress it, and place it in ./data/nuScenes/data following this structure:

   AB3DMOT/data/nuScenes/data/
   ├── samples
   ├── sweeps
   ├── v1.0-mini
   ├── v1.0-test
   └── v1.0-trainval

2. Convert to KITTI format: Run the export script to convert raw nuScenes data into the KITTI format for ground truth:

python3 scripts/nuScenes/export_kitti.py nuscenes_gt2kitti_trk --split val
python3 scripts/nuScenes/export_kitti.py nuscenes_gt2kitti_trk --split test

This generates data at ./data/nuScenes/nuKITTI/tracking.