DOCSEARCHbyAlgolia
Submit Docs

MMDetection3D Documentation

repository·main·Indexed 27 days ago

https://github.com/open-mmlab/mmdetection3d

An open-source toolbox for 3D object detection, segmentation, and other 3D vision tasks built on the OpenMMLab foundation. It includes implementations of various models such as 3DSSD, CenterPoint, Cylinder3D, DGCNN, Dynamic Voxelization, FCAF3D, and FCOS3D, with support for datasets like KITTI, nuScenes, SemanticKITTI, S3DIS, ScanNet, and SUN RGB-D.

Tokens
114.4K
Snippets
249
Records
508
Agent score
91%

What's inside MMDetection3D

  1. Overview of Probabilistic and Geometric Depth (PGD)

    main
    PGD (Probabilistic and Geometric Depth) is a monocular 3D object detector designed to improve upon the FCOS3D baseline. It treats monocular 3D detection as an instance depth estimation problem, using local geometric constraints and a probabilistic representation to capture depth uncertainty. This approach helps mitigate the impact of inaccurate depth predictions on other 3D attributes. PGD is designed to maintain real-time efficiency while achieving high performance on KITTI and nuScenes benchmarks.

  2. Overview of PV-RCNN for 3D Object Detection

    main
    PV-RCNN (Point-Voxel Feature Set Abstraction) is a 3D object detection method designed to learn high-performance features from sparse and irregular LiDAR point clouds. It achieves this by integrating point-based and voxel-based features within a unified network architecture.

  3. Overview of 3DSSD (Point-based 3D Single Stage Object Detector)

    main
    3DSSD is a lightweight, anchor-free, single-stage 3D object detector that operates directly on points. It achieves a balance between accuracy and efficiency by abandoning upsampling layers and refinement stages used in other point-based methods. It utilizes a fusion sampling strategy during downsampling and a box prediction network featuring a candidate generation layer and an anchor-free regression head with 3D center-ness assignment. It is designed for high-speed inference (>25 FPS).

  4. Overview of PointPillars for 3D Object Detection

    main
    PointPillars is a fast encoder for 3D object detection in point clouds. It organizes point clouds into vertical columns (pillars) and uses PointNets to learn their representations. The resulting encoded features can be used with any standard 2D convolutional detection architecture. PointPillars is designed to balance speed and accuracy, achieving high performance (up to 62 Hz or 105 Hz for faster versions) on datasets like KITTI, nuScenes, Lyft, and Waymo.

  5. Overview of SA-SSD (Structure Aware Single-stage 3D Object Detection)

    main

    SA-SSD is a single-stage 3D object detector designed to improve localization precision by explicitly leveraging the structure information of 3D point clouds. It uses an auxiliary network during training to convert convolutional features back to point-level representations, which is then detached during inference to ensure no extra computational overhead. It also utilizes a part-sensitive warping operation to align classification confidences with predicted bounding boxes. The implementation is available for the KITTI dataset.

    Key features:

    • High localization precision via structure-aware backbone features.
    • Zero inference overhead (auxiliary network is detached after training).
    • Efficient part-sensitive warping for confidence alignment.
    • High performance: Ranks top on KITTI 3D/BEV leaderboards and runs at ~25 FPS during inference.

  6. Overview of SSN (Shape Signature Networks)

    main
    SSN (Shape Signature Networks) is a method for multi-class 3D object detection from point clouds. It introduces a 3D shape signature—incorporating symmetry, convex hull, and Chebyshev fitting—to improve multi-class discrimination. The architecture consists of a pyramid feature encoding part, shape-aware grouping heads, and an explicit shape encoding objective. The shape signature can act as a plug-and-play component for existing models.

  7. Overview of MMDetection3D

    main

    MMDetection3D is an open-source 3D object detection toolbox built on PyTorch, part of the OpenMMLab project. It serves as a next-generation platform for general 3D detection, supporting both single-modality and multi-modality detectors (e.g., MVXNet, VoteNet, PointPillars).

    Key capabilities include:

    • Multi-modality/Single-modality support: Out-of-the-box support for various detector architectures.
    • Indoor/Outdoor detection: Support for datasets like ScanNet, SUNRGB-D, Waymo, nuScenes, Lyft, and KITTI (including nuImages).
    • 2D Integration: Natural integration with MMDetection, allowing 300+ models and 40+ paper methods to be used.
    • High Efficiency: Optimized training speeds compared to other codebases like OpenPCDet or Det3D.
    • Extensibility: Can be used as a library to build custom projects on top of its modules.

  8. FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection Overview

    main
    FCAF3D is an anchor-free 3D object detection method designed for indoor scenes. It utilizes a voxel representation of point clouds and processes them using sparse convolutions. The method is designed to handle large-scale scenes efficiently through a single fully convolutional feed-forward pass and uses a novel parametrization for oriented bounding boxes to improve generalization.

  9. Understand the three coordinate systems in MMDetection3D

    main

    MMDetection3D utilizes three distinct coordinate systems to handle different sensor types (LiDAR, depth cameras) and dataset formats. Understanding these is crucial for correct data interpretation and model training.

    1. Camera Coordinate System: Used by most cameras.

      • x-axis: Points right.
      • y-axis: Points down (towards the ground).
      • z-axis: Points forward.

    2. LiDAR Coordinate System: Standard for most LiDAR sensors.

      • x-axis: Points forward.
      • y-axis: Points left.
      • z-axis: Points up (negative z points to the ground).

    3. Depth Coordinate System: Used by models like VoteNet and H3DNet.

      • x-axis: Points right.
      • y-axis: Points forward.
      • z-axis: Points up (negative z points to the ground).

  10. Understand model component categories in MMDetection3D

    main

    MMDetection3D models are composed of six primary component types. When customizing or developing new models, you should categorize your components into one of these types:

    • encoder: Voxel encoders or middle encoders used in voxel-based methods before the backbone (e.g., HardVFE, PointPillarsScatter).
    • backbone: Typically an FCN network used to extract feature maps (e.g., ResNet, SECOND).
    • neck: Components situated between the backbone and the head (e.g., FPN, SECONDFPN).
    • head: Components responsible for specific tasks like bbox prediction or mask prediction.
    • RoI extractor: Parts used for extracting RoI features from feature maps (e.g., H3DRoIHead, PartAggregationROIHead).
    • loss: Components within the heads used for calculating losses (e.g., FocalLoss, L1Loss, GHMLoss).

  11. SECOND: Sparsely Embedded Convolutional Detection Overview

    main

    SECOND is a voxel-based 3D convolutional network designed for LiDAR-based or RGB-D-based object detection. It utilizes an improved sparse convolution method to increase training and inference speeds, introduces a new angle loss regression for better orientation estimation, and employs a specific data augmentation approach to enhance convergence. It is optimized for high-speed inference and state-of-the-art performance on benchmarks like KITTI.

    Key features include:

    • Improved sparse convolution for speed.
    • New angle loss regression for orientation.
    • Enhanced data augmentation.
    • Support for Mixed Precision (FP16) training to reduce memory usage.

  12. Understand the 6 components of MMDetection3D models

    main

    When customizing or developing new models in MMDetection3D, the model architecture is typically composed of six functional modules. Understanding these categories is essential for implementing new components or modifying existing pipelines:

    1. Encoder: Voxel-based methods used before the backbone, such as HardVFE and PointPillarsScatter.
    2. Backbone: FCN-based networks used to extract feature maps, such as ResNet and SECOND.
    3. Neck: Modules located between the backbone and the head, such as FPN and SECONDFPN.
    4. Head: Task-specific modules used for predictions, such as bounding box prediction or mask prediction.
    5. RoI Extractor: Modules used to extract RoI features from feature maps, such as H3DRoIHead and PartAggregationROIHead.
    6. Loss: Modules within the heads used to calculate loss, such as FocalLoss, L1Loss, and GHMLoss.