CenterPoint 3D Object Detection and Tracking

CenterPoint is a framework for 3D Object Detection and Tracking that represents objects as points in a bird-eye view (BEV) rather than using axis-aligned or rotated bounding boxes.

Core Workflow:

• Detection: Uses a keypoint detector to find object centers in a BEV heatmap, then regresses attributes like 3D size, 3D orientation, and velocity. A second stage refines these estimates using point features.
• Tracking: Simplifies 3D object tracking to a greedy closest-point matching approach.

Key Characteristics:

• Simple: Uses standard 3D point cloud encoders with convolutional layers in the head.
• Fast and Accurate: Capable of high performance (e.g., 71.9 mAPH on Waymo) at usable frame rates (11+ FPS).
• Extensible: Can be used as a replacement for anchor-based detectors in novel algorithms.

The final CenterPoint models often use a two-stage training process. For example, to train a two-stage CenterPoint-Voxel model:

1. Train the one-stage model using a configuration like configs/waymo/voxelnet/waymo_centerpoint_voxelnet_3x.py.
2. Train the second-stage module using a configuration like configs/waymo/voxelnet/two_stage/waymo_centerpoint_voxelnet_two_stage_bev_5point_ft_6epoch_freeze.py.

To perform benchmark evaluation or training, follow these steps:

1. Prepare Data: Follow the instructions in docs/GETTING_START.md to prepare your dataset.
2. Reproduce Results: Use the instructions in docs/GETTING_START.md to run detection and tracking.
3. Configurations: All detection configurations are located in the configs directory.

If you want to extend the project, refer to docs/DEVELOP.md for guidance on:

• Training CenterPoint on a new dataset.
• Using CenterPoint for a new task.
• Implementing a new network architecture within the CenterPoint framework.

To prepare the Waymo dataset, follow these steps:

1. Organize Directory Structure: Ensure your raw data is organized as follows:

└── WAYMO_DATASET_ROOT
       ├── tfrecord_training       
       ├── tfrecord_validation   
       └── tfrecord_testing 

2. Convert TFRecords to Pickle: Use det3d/datasets/waymo/waymo_converter.py to convert the raw records into pickle files. Replace WAYMO_DATASET_ROOT with your actual path.

3. Symlink Data: Create a data/Waymo symlink pointing to your dataset root.

4. Generate Info Files: Use tools/create_data.py with the waymo_data_prep argument. You can generate info files for either one-sweep (--nsweeps=1) or two-sweep (--nsweeps=2) models.

You can perform distributed testing, single-GPU testing with speed metrics, or local evaluation.

• Distributed Testing (4 GPUs): Use tools/dist_test.py with --nproc_per_node=4.
• Single GPU Testing (with speed test): Use the --speed_test flag to see inference time. This generates a my_preds.bin file.
• Local Evaluation: To evaluate a subset locally, use det3d/datasets/waymo/waymo_common.py to generate ground truth prediction bin files.
• Test Set: Append the --testset flag to the testing command to run on the test split.

To extend the model architecture, add your components to the following directory structure:

• 3D Backbones: Add to det3d/models/backbones.
• 2D Backbones: Add to det3d/models/necks.
• Two-stage Refinement Modules: Add to det3d/models/second_stage.

Before using CenterPoint with the Waymo dataset, ensure you have followed the general installation guide in INSTALL.md. Additionally, you must install the waymo-open-dataset-tf-1-15-0 package. It is recommended to use a dedicated conda environment.

conda activate centerpoint 
pip install waymo-open-dataset-tf-1-15-0==1.2.0 

To generate predictions for the official nuScenes test set, follow these steps:

1. Organize Data: Ensure your data/nuScenes directory includes the v1.0-test folder containing samples, sweeps, maps, metadata, and the infos_test_10sweeps_withvelo.pkl file.
2. Download Checkpoint: Download the centerpoint_voxel_1440_flip checkpoint and save it to work_dirs/nusc_0075_flip.
3. Run Inference: Execute the following command to generate predictions using the --testset flag:

python tools/dist_test.py configs/nusc/voxelnet/nusc_centerpoint_voxelnet_0075voxel_fix_bn_z_flip.py --work_dir work_dirs/nusc_centerpoint_voxelnet_dcn_0075voxel_flip_testset --checkpoint work_dirs/nusc_0075_flip/voxelnet_converted.pth --testset

To install spconv, you must first install libboost-all-dev via apt. Then, clone the spconv repository, checkout the specific commit 7342772, build the wheel, and install it via pip.

sudo apt-get install libboost-all-dev
git clone https://github.com/traveller59/spconv.git --recursive
cd spconv && git checkout 7342772
python setup.py bdist_wheel
cd ./dist && pip install *

To develop new tasks based on CenterPoint's detection results, follow these steps:

1. Tracking Algorithms: For tracking development, refer to the specific guides for NUSC and WAYMO.
2. Motion Prediction: For advanced tasks like motion prediction, you may need to store the final BEV (Bird's Eye View) feature map. This is computed in det3d/models/necks/rpn.py.
3. Data Pipelines: You must add files to det3d/datasets/pipelines/preprocess.py to specify how data is generated during training and inference.
4. Data Loading: You may need to modify the following components to support your task's data requirements:
   • The collate function in det3d/torchie/parallel/collate.py.
   • The data loading function in det3d/torchie/trainer/trainer.py.

To access pretrained models for Waymo 3D Detection, you must send an email to yintianwei@utexas.edu including:

1. Your name
2. Your institute
3. A screenshot of your Waymo dataset registration confirmation email
4. Your intended usage

Note: The Waymo open dataset is under a strict non-commercial license. Models will not be shared for profit-oriented activities.