BEVFusion

To set up the environment using Docker, ensure nvidia-docker is installed.

1. Build the image:

cd docker && docker build . -t bevfusion

2. Run the container (mapping your local data directory to /dataset inside the container):

nvidia-docker run -it -v `pwd`/../data:/dataset --shm-size 16g bevfusion /bin/bash

3. Inside the container, clone the repo and install custom CUDA extensions:

cd home && git clone https://github.com/mit-han-lab/bevfusion && cd bevfusion
python setup.py develop

Note: It is recommended to perform data preparation outside the Docker container. Use an absolute path for the dataset directory.

The following library versions are required for BEVFusion:

• Python: >= 3.8, < 3.9
• OpenMPI: 4.0.4
• mpi4py: 3.0.3 (Required for torchpack)
• Pillow: 8.4.0
• PyTorch: >= 1.9, <= 1.10.2
• mmcv: 1.4.0
• mmdetection: 2.20.0
• Other dependencies: tqdm, torchpack, nuscenes-dev-kit

To install the BEVFusion codebase locally, ensure you have the required dependencies installed first. Then, run the following command from the repository root:

python setup.py develop

To use the nuScenes dataset, download and preprocess it following the mmdetection3d instructions.

Important:

1. Download both the detection dataset and the map extension (required for BEV map segmentation).
2. It is recommended to re-generate the .pkl info files using the BEVFusion codebase rather than using files from standard mmdetection3d, due to coordinate system differences.

Expected directory structure:

mmdetection3d
├── mmdet3d
├── tools
├── configs
├── data
│   ├── nuscenes
│   │   ├── maps
│   │   ├── samples
│   │   ├── sweeps
│   │   ├── v1.0-test
│   │   ├── v1.0-trainval
│   │   ├── nuscenes_database
│   │   ├── nuscenes_infos_train.pkl
│   │   ├── nuscenes_infos_val.pkl
│   │   ├── nuscenes_infos_test.pkl
│   │   ├── nuscenes_dbinfos_train.pkl

To evaluate pretrained models, first download the checkpoints:

./tools/download_pretrained.sh

Then, use torchpack dist-run to run the evaluation. The syntax is: torchpack dist-run -np [number of gpus] python tools/test.py [config file path] pretrained/[checkpoint name].pth --eval [evaluation type]

Examples:

• 3D Object Detection (Camera+LiDAR):

torchpack dist-run -np 8 python tools/test.py configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/convfuser.yaml pretrained/bevfusion-det.pth --eval bbox

• BEV Map Segmentation:

torchpack dist-run -np 8 python tools/test.py configs/nuscenes/seg/fusion-bev256d2-lss.yaml pretrained/bevfusion-seg.pth --eval map