DOCSEARCHbyAlgolia
Submit Docs

Point Transformer V3 (PTv3) Documentation

repository·main·Indexed 23 days ago

https://github.com/pointcept/pointtransformerv3

A high-performance 3D point cloud perception model designed for speed and simplicity. Part of the Pointcept ecosystem, PTv3 supports indoor and outdoor semantic segmentation on datasets such as ScanNet, S3DIS, nuScenes, and Waymo. It features optional FlashAttention support for optimized performance and can be used either within the full Pointcept framework or integrated as a standalone model into custom projects.

Tokens
1.9K
Snippets
5
Records
7
Agent score
34%

What's inside Point Transformer V3 (PTv3)

  1. Perform S3DIS 6-fold cross validation

    main

    To perform 6-fold cross validation on the S3DIS dataset, follow these steps:

    1. Configure Splits: Modify the default configs by changing data.train.split, data.val.split, and data.test.split to evaluate on Area_1 through Area_6 respectively.
    2. Train and Collect: Train and evaluate the model on each split. Gather the resulting .pth files (located in exp/s3dis/EXP_NAME/result/Area_x.pth) into a single folder named RECORD_FOLDER.
    3. Run Validation Script: Execute the tools/test_s3dis_6fold.py script pointing to your record folder.
    export PYTHONPATH=./
python tools/test_s3dis_6fold.py --record_root ${RECORD_FOLDER}

  2. Install Point Transformer V3

    main

    To install PTv3, you must first set up a base Conda environment and install the necessary dependencies. PTv3 relies on FlashAttention for optimal performance, which requires CUDA 11.6 or higher. If you cannot meet these requirements, you must disable Flash Attention in the model configuration.

    Recommended Requirements

    • Ubuntu: 20.04 and above
    • CUDA: 11.6 and above
    • PyTorch: 1.12.0 and above

    Minimum Requirements (with Flash Attention disabled)

    • Ubuntu: 18.04 and above
    • CUDA: 11.3 and above
    • PyTorch: 1.10.0 and above
    conda create -n pointcept python=3.8 -y
conda activate pointcept
conda install ninja -y
# Using development versions: CUDA 11.8 and PyTorch 2.1.0
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=11.8 -c pytorch -c nvidia
conda install h5py pyyaml -c anaconda -y
conda install sharedarray tensorboard tensorboardx yapf addict einops scipy plyfile termcolor timm -c conda-forge -y
conda install pytorch-cluster pytorch-scatter pytorch-sparse -c pyg -y
pip install torch-geometric

cd libs/pointops
python setup.py install
cd ../..

# Install spconv (match your local cuda version)
pip install spconv-cu118

# Optional visualization
pip install open3d

  3. Quick Start: Two running scenarios

    main

    There are two ways to use PTv3 depending on your project structure:

    1. Pointcept-driven (Recommended)

    Use this if you want to use the full Pointcept framework. Clone the Pointcept repository and use the provided training scripts.

    2. Custom-framework-driven

    Use this to integrate PTv3 into an existing project without the full Pointcept framework. Clone the PTv3 repository and copy the core model files into your project directory. You must align your input dictionary with the one defined in model.py to ensure the model returns the correct encoded features.

    # Scenario 1: Pointcept-driven
git clone https://github.com/Pointcept/Pointcept.git
sh scripts/train.sh -p ${INTERPRETER_PATH} -g ${NUM_GPU} -d ${DATASET_NAME} -c ${CONFIG_NAME} -n ${EXP_NAME}

# Scenario 2: Custom-framework-driven
git clone https://github.com/Pointcept/PointTransformerV3.git
cp model.py ${PATH_TO_YOUR_PROJECT}
cp -r serialization ${PATH_TO_YOUR_PROJECT}

  4. Configure Flash Attention for PTv3

    main

    Flash Attention is highly recommended for PTv3 but is optional. If your environment does not support CUDA 11.6 or higher, you must disable it.

    To disable Flash Attention:

    1. Set the model parameter enable_flash to false.
    2. Reduce enc_patch_size and dec_patch_size to a lower value (e.g., 128).

    Note on Accuracy: Flash Attention forces the disabling of RPE (Relative Positional Encoding) and reduces accuracy to fp16. If you require RPE or higher precision, disable enable_flash and manually adjust enable_rpe, upcast_attention, and upcast_softmax.

  5. Train PTv3 models for Outdoor Semantic Segmentation

    main

    PTv3 models can be trained on outdoor datasets including nuScenes and Waymo using the scripts/train.sh script.

    Training Scenarios:

    • nuScenes (Scratch): Use 4 GPUs.
    • Waymo (Scratch): Use 4 GPUs.

    Note: Model weights trained with the Waymo Open Dataset cannot be released due to regulations.

    # Scratched nuScenes
sh scripts/train.sh -g 4 -d nuscenes -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base

# Scratched Waymo
sh scripts/train.sh -g 4 -d waymo -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base

  6. Train PTv3 models for Indoor Semantic Segmentation

    main

    You can train Point Transformer V3 (PTv3) models on various indoor datasets like ScanNet, ScanNet200, and S3DIS using the scripts/train.sh script. The training can be performed from scratch or using PPT (joint training with additional data like Structured3D).

    Common Training Scenarios:

    • ScanNet (Scratch): Use 4 GPUs.
    • ScanNet (PPT joint training): Use 8 GPUs for joint training on ScanNet + Structured3D.
    • ScanNet200 (Scratch): Use 4 GPUs.
    • S3DIS (Scratch): Uses RPE (Relative Position Encoding) and serves as an example for disabling flash attention. Use 4 GPUs.
    • S3DIS (PPT joint training): Joint training on ScanNet + S3DIS + Structured3D. Use 8 GPUs.
    # Scratched ScanNet
sh scripts/train.sh -g 4 -d scannet -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base

# PPT joint training (ScanNet + Structured3D) and evaluate in ScanNet
sh scripts/train.sh -g 8 -d scannet -c semseg-pt-v3m1-1-ppt-extreme -n semseg-pt-v3m1-1-ppt-extreme

# Scratched ScanNet200
sh scripts/train.sh -g 4 -d scannet200 -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base

# Scratched S3DIS
sh scripts/train.sh -g 4 -d s3dis -c semseg-pt-v3m1-0-rpe -n semseg-pt-v3m1-0-rpe

# PPT joint training (ScanNet + S3DIS + Structured3D) and evaluate in ScanNet
sh scripts/train.sh -g 8 -d s3dis -c semseg-pt-v3m1-1-ppt-extreme -n semseg-pt-v3m1-1-ppt-extreme

  7. Reference: train.sh CLI arguments

    main

    The scripts/train.sh script is used to initiate training. Based on the usage examples, it accepts the following arguments:

    • -g: Number of GPUs to use.
    • -d: Dataset name (e.g., scannet, scannet200, s3dis, nuscenes, waymo).
    • -c: Configuration file name/identifier.
    • -n: Experiment name.