DOCSEARCHbyAlgolia
Submit Docs

BEVDet: Multi-Camera 3D Object Detection Framework

repository·dev3.0·Indexed 23 days ago

https://github.com/huangjunjie2017/bevdet

A high-performance 3D object detection framework in Bird-Eye-View (BEV) supporting temporal modeling (4D), depth supervision, and stereo implementations for the nuScenes dataset. The repository includes implementations of various 3D detection and segmentation models, including 3DSSD (anchor-free single-stage detector), CenterPoint, DGCNN for S3DIS segmentation, Dynamic Voxelization for multi-view fusion, and FCAF3D (fully convolutional anchor-free 3D detection).

Tokens
113.4K
Snippets
216
Records
433
Agent score
81%

What's inside BEVDet

  1. Overview of Probabilistic and Geometric Depth (PGD)

    dev3.0

    PGD (Probabilistic and Geometric Depth) is a monocular 3D object detector designed to improve upon the FCOS3D baseline. It addresses the challenge of inaccurate instance depth estimation in monocular 3D detection by incorporating local geometric constraints and a probabilistic representation to capture depth uncertainty.

    Key features include:

    • Geometric Constraints: Uses local geometric relations to facilitate depth estimation.
    • Probabilistic Depth Representation: Captures uncertainty in depth predictions to identify confident predictions and guide depth propagation.
    • Efficiency: Maintains real-time efficiency while achieving state-of-the-art results on KITTI and nuScenes benchmarks.

    Note: The current preliminary release supports base models with local geometric constraints and probabilistic depth representation. The geometric graph component is planned for a future release.

  2. Overview of MonoFlex: Flexible Monocular 3D Object Detection

    dev3.0
    MonoFlex is a framework for monocular 3D object detection designed to handle diverse object distributions, specifically addressing the challenge of truncated objects. It achieves this by decoupling the feature map edges for long-tail truncated objects and using an uncertainty-guided ensemble for depth estimation (combining directly regressed depth with depths solved from keypoints).

  3. Overview of Group-Free 3D Object Detection

    dev3.0
    Group-Free 3D Object Detection via Transformers is a method for directly detecting 3D objects from 3D point clouds without a manual point grouping step. Instead of assigning points to object candidates using hand-crafted schemes, it uses a Transformer-based attention mechanism to automatically learn the contribution of each point to an object's feature. This approach aims to improve performance by avoiding inaccurate point assignments common in traditional grouping methods.

  4. Overview of SA-SSD (Structure Aware Single-stage 3D Object Detection)

    dev3.0

    SA-SSD is a single-stage 3D object detector designed to improve localization precision by explicitly leveraging the structure information of 3D point clouds.

    Key features include:

    • Auxiliary Network: During training, an auxiliary network converts convolutional features back to point-level representations. This network is jointly optimized with two point-level supervisions to guide the backbone to be aware of object structures.
    • Zero Inference Overhead: The auxiliary network can be detached after training, meaning it adds no extra computation during inference.
    • Part-sensitive Warping: An efficient operation that aligns classification confidences with predicted bounding boxes to resolve discordance.
    • Performance: Achieves high ranking on KITTI 3D/BEV detection leaderboards and runs at approximately 25 FPS during inference.

  5. Overview of FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection

    dev3.0
    FCAF3D is an indoor 3D object detection method that uses a voxel representation of point clouds and processes them with sparse convolutions. It is designed to be anchor-free and fully convolutional, allowing it to handle large-scale scenes via a single feed-forward pass. It achieves state-of-the-art results on datasets like ScanNet V2, SUN RGB-D, and S3DIS by using a novel parametrization of oriented bounding boxes to eliminate prior geometric assumptions.

  6. Overview of SECOND: Sparsely Embedded Convolutional Detection

    dev3.0
    SECOND is a LiDAR-based 3D object detection method that utilizes improved sparse convolution to increase the speed of training and inference. It addresses common issues in voxel-based 3D convolutional networks, such as slow inference speeds and low orientation estimation performance, by introducing an improved sparse convolution method, a new angle loss regression, and a new data augmentation approach. The implementation provides state-of-the-art results on KITTI benchmarks.

  7. Overview of 3DSSD: Point-based 3D Single Stage Object Detector

    dev3.0

    3DSSD is a lightweight, anchor-free, single-stage 3D object detector designed for high efficiency and accuracy. Unlike voxel-based methods, it operates directly on points and avoids heavy upsampling layers and refinement stages to reduce computational costs. It uses a fusion sampling strategy during downsampling and a box prediction network featuring a candidate generation layer and an anchor-free regression head with 3D center-ness assignment.

    Key characteristics:

    • Efficiency: Achieves inference speeds of over 25 FPS.
    • Paradigm: Single-stage, anchor-free framework.
    • Datasets: Evaluated on KITTI and nuScenes.

  8. Overview of PointPillars for Point Cloud Object Detection

    dev3.0

    PointPillars is a fast encoder designed for object detection in point clouds, specifically optimized for autonomous driving applications. It organizes point clouds into vertical columns called 'pillars' and uses PointNets to learn their representations. The resulting encoded features are compatible with standard 2D convolutional detection architectures.

    Key characteristics include:

    • High Speed: The full detection pipeline can run at 62 Hz, offering a 2-4x runtime improvement over previous methods. A faster version can reach 105 Hz.
    • High Accuracy: Outperforms previous encoders in both speed and accuracy, even competing with fusion-based methods using only LiDAR data.
    • Versatility: The encoded features can be used with any standard 2D convolutional detection architecture.

  9. Overview of FCOS3D

    dev3.0
    FCOS3D is an anchor-free, one-stage monocular 3D object detector adapted from the 2D FCOS detector. It is built on top of mmdetection and mmdetection3d and is designed for monocular vision tasks, specifically optimized for the nuScenes dataset. It works by transforming 7-DoF 3D targets into the image domain and decoupling them into 2D and 3D attributes.

  10. Overview of SSN (Shape Signature Networks)

    dev3.0

    SSN (Shape Signature Networks) is a method for multi-class 3D object detection from point clouds. It introduces a 3D shape signature—incorporating symmetry, convex hull, and Chebyshev fitting—to better exploit shape information, making the model more robust to noise and improving multi-class discrimination.

    Key components include:

    • Pyramid feature encoding
    • Shape-aware grouping heads
    • Explicit shape encoding objective

    The shape signature is designed as a plug-and-play component that can be integrated into existing architectures like PointPillars.

  11. Overview of MMDetection3D

    dev3.0

    MMDetection3D is an open-source 3D object detection toolbox based on PyTorch. It is part of the OpenMMLab project and serves as a next-generation platform for 3D detection tasks.

    Key capabilities include:

    • Multi-modal & Single-modal Support: Supports detectors like MVXNet, VoteNet, and PointPillars.
    • Diverse Dataset Support: Compatible with indoor and outdoor datasets including ScanNet, SUNRGB-D, Waymo, nuScenes (including nuImages), Lyft, and KITTI.
    • 2D Integration: Seamlessly integrates with MMDetection, allowing the use of 300+ 2D models and 40+ algorithms.
    • High Performance: Optimized for faster training speeds compared to other libraries.

    Requirements:

    • PyTorch 1.3 or higher.

  12. ImVoxelNet: Image to Voxels Projection for 3D Object Detection

    dev3.0
    ImVoxelNet is a fully convolutional method for 3D object detection using posed monocular or multi-view RGB images. It is designed as an end-to-end optimization problem and can handle a variable number of monocular images per input, making it suitable for both indoor and outdoor scenes. It supports monocular detection (e.g., on KITTI) and multi-view detection (e.g., on nuScenes and SUN RGB-D).