Open3D

Open3D provides real-time 3D Gaussian Splatting (3DGS) rendering using a GPU compute pipeline that integrates with the Filament-based visualization engine.

Key Features:

• Hybrid Rendering: Uses a compute pipeline for splats alongside Filament for mesh geometry.
• Occlusion: Employs a shared depth texture to allow mesh geometry to occlude Gaussian splats correctly.
• Multi-Object Support: Supports multiple Gaussian scenes within a single environment.
• Offscreen Rendering: Supports RenderToImage and RenderToDepthImage for non-interactive captures.

Supported Platforms:

• Linux: X11/GLX (including Wayland via XWayland).
• Windows: WGL.
• macOS: Metal.

Open3D-ML is an extension of the Open3D core library specifically designed for 3D machine learning tasks. It provides specialized machine learning tools and algorithms for processing 3D data, building upon the existing Open3D framework.

You can find the dedicated repository at: https://github.com/isl-org/Open3D-ML.

The open3d.ml.torch.layers module provides specialized PyTorch layers designed for 3D data processing and machine learning tasks. These layers are optimized for handling sparse data, voxelized structures, and geometric operations common in 3D deep learning pipelines.

Available layer types include:

• Convolutional Layers: SparseConv, SparseConvTranspose, and ContinuousConv for processing sparse or continuous 3D signals.
• Search Layers: KNNSearch, RadiusSearch, and FixedRadiusSearch for efficient geometric neighbor queries.
• Voxel Operations: VoxelPooling for aggregating features within voxel grids.

The open3d.ml.torch.ops module provides a collection of specialized PyTorch operations designed for 3D machine learning tasks. These operations are optimized for processing 3D data (such as point clouds) within a PyTorch computational graph, enabling efficient GPU-accelerated workflows for tasks like spatial searching, voxelization, and convolution.

Available functions include:

• Spatial & Search: build_spatial_hash_table, fixed_radius_search, knn_search, radius_search, invert_neighbors_list.
• Voxel & Pooling: voxelize, voxel_pooling, reduce_subarrays_sum.
• Convolution: continuous_conv, continuous_conv_transpose.
• Filtering: nms (Non-Maximum Suppression).

The open3d.ml.tf.layers module provides specialized TensorFlow layers designed for 3D data processing and machine learning tasks. These layers are optimized for handling sparse and continuous 3D data structures commonly used in point cloud processing.

Available layer classes include:

• Convolutional Layers:
  • ContinuousConv: For continuous 3D convolutions.
  • SparseConv: For sparse 3D convolutions.
  • SparseConvTranspose: For sparse transposed (deconvolutional) operations.
• Search & Neighborhood Layers:
  • FixedRadiusSearch: For finding neighbors within a fixed radius.
  • RadiusSearch: For radius-based neighbor searches.
  • KNNSearch: For K-Nearest Neighbor searches.
• Pooling Layers:
  • VoxelPooling: For voxel-based pooling operations.

The Open3D reconstruction system follows a 4-step pipeline to transform an RGBD sequence into a complete 3D mesh model:

1. Make Fragments: Builds local geometric surfaces (fragments) from short subsequences of the input RGBD sequence using RGBD odometry, multiway registration, and RGBD integration.
2. Register Fragments: Aligns fragments in a global space to detect loop closures using global registration, ICP registration, and multiway registration.
3. Refine Registration: Tightens the rough alignments using ICP registration and multiway registration.
4. Integrate Scene: Generates the final mesh model by integrating the RGB-D images.

A Voxel Block Grid is a data structure designed to represent 3D scenes using a globally sparse and locally dense approach.

• Globally Sparse: The 3D space is divided into blocks organized in a hash map by 3D coordinates. Only blocks containing surfaces are stored, which saves memory since surfaces occupy a small portion of total 3D space.
• Locally Dense: Each block is further divided into dense voxels that can be accessed by array indices. This preserves data locality in memory, which is more efficient than using a hash map for every individual voxel.

This structure is optimized for high-performance GPU execution, though it can also run on the CPU (at slower speeds).

Starting from v0.19, Open3D provides an experimental SYCL backend for cross-platform GPU support. This allows Open3D operations to run on various GPUs, including integrated and discrete GPUs from Intel, Nvidia, and AMD.

Key Characteristics:

• Tensor API Support: Many Tensor and Tensor Geometry operations are supported on SYCL devices.
• Device Mapping: Typically, SYCL:0 refers to the available GPU, and SYCL:1 refers to the CPU (if a GPU is present).
• JIT Compilation: Kernels are distributed as SPIR-V IR and are JIT-compiled to the target GPU's native ISA at runtime. The first execution of a kernel will be slower due to this compilation.
• Limitations: SYCL does not implement CUDA NPP/IPP image filters (e.g., Image::Filter*, Resize, PyrDown). Use CPU preprocessing for these tasks.