TorchSparse Documentation

If you prefer not to use pre-built wheels, you can install TorchSparse from source. You can either clone the repository and use setup.py or install directly via pip from the GitHub URL.

Option 1: From a cloned repository

python setup.py install

Option 2: Directly via pip

pip install git+https://github.com/mit-han-lab/torchsparse.git

To run an evaluation using the plugin, use the following command structure in your Conda environment:

python <test_file_path> <cfg_path> <torchsparse_model_path> <cfg_options> --task lidar_det

Arguments

• <test_file_path>: The path to tools/test.py in the mmdet3d repository.
• <cfg_path>: The path to the MMDetection3D model config file.
• <torchsparse_model_path>: The path to your converted TorchSparse checkpoint.
• <cfg_options>: Required MMDet3D configuration options to switch model layers to the TorchSparse plugin versions.

For Linux systems, it is recommended to use pre-built wheels to avoid complex building processes. The installation script automatically detects your PyTorch and CUDA versions to find the matching wheel.

Prerequisites

1. PyTorch: Version 1.9.0 or higher must be installed.

   python -c "import torch; print(torch.__version__)"

2. CUDA: If using GPUs, PyTorch must be installed with CUDA support.

   python -c "import torch; print(torch.version.cuda)"

Installation Command

Run the following command to execute the automatic installation script:

python -c "$(curl -fsSL https://raw.githubusercontent.com/mit-han-lab/torchsparse/master/install.py)"

To use TorchSparse with MMDetection3D, follow these installation steps:

1. Install MMDetection3D: Follow the official MMDetection3D documentation.
2. Prepare Datasets: Pre-process your datasets according to MMDetection3D dataset preparation guides.
3. Install TorchSparse: Install the core TorchSparse library.
4. Install the Plugin:
   • Clone this repository.
   • Navigate to examples/mmdetection3d.
   • Run pip install -v -e ..

Note: For model evaluation, ensure you update the data root in the original MMDetection3D model config to the absolute path of your dataset root.

Because TorchSparse dimensions differ from SpConv, you must convert model weights. You can use the convert_weights_cmd() CLI tool in converter.py or the convert_weights() API.

Parameters

• ckpt_before: Path to the input SpConv checkpoint file.
• ckpt_after: Path where the converted TorchSparse checkpoint will be saved.
• cfg_path: Path to the model's configuration file.
• v_spconv: The version of SpConv used in the original model (1 or 2).
• framework: The framework type, either 'openpc' or 'mmdet3d' (defaults to 'mmdet3d').

Example CLI Usage

python examples/converter.py --ckpt_before ../OpenPCDet/models/SECOND/second_7862.pth --cfg_path ../OpenPCDet/tools/cfgs/kitti_models/second.yaml --ckpt_after ./converted/SECOND/second_7862.pth --v_spconv 1 --framework openpc

To evaluate models using the TorchSparse backend, use the standard test.py provided by MMDetection3D or OpenPCDet, but follow these specific steps:

1. Use Converted Weights: Provide the ckpt_after file generated by the converter as the model weights.
2. Activate Plugins: You must manually import the plugins in the test script to activate them:
   • For MMDet3D: Add import ts_plugin to mmdetection3d/tools/test.py.
   • For OpenPCDet: Add import pcdet_plugin to OpenPCDet/tools/test.py.
3. Replace Layers:
   • MMDet3D: You must provide arguments to replace certain layers with TorchSparse versions (refer to examples/mmdetection3d/demo.ipynb for implementation details).
   • OpenPCDet: Use the modified configuration files located in examples/openpcdet/cfgs (refer to examples/openpcdet/demo.ipynb for usage).

To use TorchSparse with OpenPCDet, you must install the plugin following these steps:

1. Ensure Conda, OpenPCDet, and TorchSparse are already installed.
2. Pre-process your datasets according to OpenPCDet requirements.
3. Clone the torchsparse repository.
4. Set the PCDET_BASE environment variable to the absolute path of your OpenPCDet installation.
5. Navigate to the examples/openpcdet directory and install the plugin in editable mode:

pip install -v -e .

If the automatic installation script or the PyPI server fails, you can manually download the appropriate wheels from the official wheel index.

Wheel versioning follows the pattern: {version}+torch{pytorch_version}cu{cuda_version}.

Example: For PyTorch 1.11.0 and CUDA 11.5, the version string would be 2.1.0+torch111cu115.

Download wheels here: http://pypi.hanlab.ai/simple/torchsparse

The simplest way to run demos is using the pre-configured Docker image ioeddk/torchsparse_plugin_demo:latest. This image includes MMDetection3D, OpenPCDet, TorchSparse, plugins, and PyTorch (based on NVIDIA CUDA 12.1).

Note: Datasets are not included in the image and must be bind-mounted when starting the container.

To start the container with KITTI and NuScenes datasets mounted, use:

docker run -it --gpus all --mount type=bind,source=<kitti_dataset_root>,target=/root/data/kitti --mount type=bind,source=<nuscenes_dataset_root>,target=/root/data/nuscenes ioeddk/torchsparse_plugin_demo:latest

Once inside the container, you can run the demos by opening the following notebooks:

• MMDetection3D: /root/repo/torchsparse-dev/examples/mmdetection3d/demo.ipynb
• OpenPCDet: /root/repo/torchsparse-dev/examples/openpcdet/demo.ipynb

The plugin is not automatically loaded by MMDetection3D. You must manually activate it by adding an import statement to the test script.

Action: In mmdetection3d/tools/test.py, add import ts_plugin as the last import statement in the file.

Because TorchSparse dimensions differ from SpConv, you must convert pretrained weights to use the TorchSparse backend. Use the examples/converter.py script.

Function Signature: convert_weights(ckpt_before: str, ckpt_after: str, cfg_path: str, v_spconv: int = 1, framework: str = "mmdet3d")

Arguments:

• ckpt_before: Path to the original pretrained checkpoint (e.g., from MMDetection3d or OpenPCDet model Zoo).
• ckpt_after: Output path for the converted checkpoint.
• cfg_path: Path to the model's config file (required to instantiate the model and locate Sparse Convolution layers).
• v_spconv: The SpConv version used by the original model (1 or 2).
• framework: The target framework, either mmdet3d or openpc.