DOCSEARCHbyAlgolia
Submit Docs

Kaolin

repository·master·Indexed 24 days ago

https://github.com/nvidiagameworks/kaolin

NVIDIA's PyTorch library of GPU-optimized modules for 3D deep learning research. It provides tools for physics simulation (Simplicits), differentiable rendering (DIB-R, nvdiffrast, easy_render PBR), 3D Gaussian splatting, and structured point clouds (SPC). The library includes utilities for USD pipeline integration, GPU-based 3D representation conversions, and visualization tools such as Kaolin Dash3D.

Tokens
19.3K
Snippets
25
Records
137
Agent score
84%

What's inside Kaolin

  1. Overview of Sample Meshes

    master
    The sample_data/meshes/ directory contains various sample meshes used for stress-testing and prototyping within Kaolin. These meshes are simple test cases and are not intended for high-quality 3D production. They cover various formats (OBJ, USDA, USDC, GLTF), shading types (flat vs. smooth), and complexities (single mesh, multi-material, multi-object, and volumetric data).

  2. Overview of Kaolin functional modules

    master

    Kaolin provides several specialized modules for 3D deep learning research:

    • kaolin.ops: Operators for efficient processing of batched 3D models and tensors, including conversions between 3D representations, primitives, batching of heterogeneous data, and functions for meshes and voxel grids.
    • kaolin.io: I/O functionality for interacting with files, including importers/exporters for formats like .obj and .usd, and utilities for preprocessing and caching datasets.
    • kaolin.metrics: Functions to compute distances and losses, such as point_to_mesh distance, Chamfer distance, IoU, or Laplacian smoothing.
    • kaolin.render: Differentiable rendering functions including DIB-R rasterization, camera projection/translation/rotation, lighting, and textures.
    • kaolin.visualize: Utilities for writing 3D checkpoints for visualization, including a timelapse exporter compatible with the Omniverse Kaolin App.
    • kaolin.utils: General utility functions for research scripts, such as tensor information display and seed fixing.
    • kaolin.non_commercial: Features restricted to non-commercial research and evaluation purposes (e.g., FlexiCubes).

  3. Overview of Kaolin features

    master

    Kaolin is a PyTorch library for 3D deep learning research providing:

    • Physics (Simplicits): Representation-agnostic, warp-accelerated simulation for meshes, splats, and point clouds.
    • 3D Gaussian Splats: Support for GaussianSplatModel, PLY/USD I/O, densification, and gsplat camera converters.
    • Differentiable Rendering: Support for DIB-R, nvdiffrast, and easy_render PBR.
    • Structured Point Clouds (SPC): GPU octree acceleration with ray tracing and feature grids.
    • USD Pipeline: Import/export for meshes, point clouds, gaussians, and physics materials using a custom schema.
    • Conversions: Fast GPU-based conversions between meshes, voxel grids, point clouds, and gaussians.
    • Quaternions: Differentiable quaternion and rigid-transform utilities.
    • Surface Meshes: Batched SurfaceMesh containers with auto-computed attributes.
    • Visualization: Jupyter 3D viewer, timelapse checkpoints, and a web client-server framework (kaolin/visualize/dash).

  4. Overview of kaolin.io module

    master

    The kaolin.io directory provides functionalities for interacting with 3D data files. It includes importers and exporters for various 3D formats, dataset preprocessing utilities, and material definitions.

    Key submodules include:

    • Format Importers/Exporters: .obj, .usd(a), .gltf, .off, .ply, and Gaussian Splatting formats.
    • Dataset Management: kaolin.io.dataset for caching data and preprocessing whole datasets.
    • Materials: kaolin.io.materials for defining materials used throughout the Kaolin library.
    • Specialized Data: Support for specific datasets like shapenet, modelnet, and shrec, as well as rendering utilities.

  5. Overview of Kaolin Library features

    master

    NVIDIA Kaolin is a PyTorch-based library designed to accelerate 3D deep learning research. It provides GPU-optimized operations for various 3D representations and includes the following capabilities:

    • Differentiable Rendering: Modular differentiable rendering, differentiable camera API, and differentiable lighting (using spherical harmonics and spherical gaussians).
    • 3D Representations & Conversions: Fast conversions between different 3D representations and a convenient batched mesh container.
    • Acceleration Structures: Structured Point Clouds (a powerful octree acceleration structure).
    • Math & Physics: Quaternion operations and representation-agnostic physics simulation.
    • Data & Visualization: Data loading, 3D checkpoints, and an interactive 3D visualizer for Jupyter notebooks.

  6. Use Apache-licensed Kaolin modules

    master

    Kaolin is released under the Apache License 2.0. A standard import kaolin provides access to the full Apache-licensed library.

    Note on Legacy Modules: The kaolin/non_commercial/ package is for legacy support only (e.g., for backward compatibility with older FlexiCubes copies). For all new development, use the Apache-licensed modules located in:

    • kaolin/ops/
    • kaolin/rep/
    • And other standard locations within the package tree.

  9. Kaolin System Requirements

    master

    To use the full functionality of Kaolin (including high-performance C++ and CUDA operations), ensure your system meets these requirements:

    • OS: Linux, Windows, or macOS (Note: macOS is CPU-only)
    • Python: >= 3.9, <= 3.12
    • CUDA: >= 10.0 (with nvcc installed)
    • PyTorch: >= 2.0, <= 2.5.1 (Note: Source builds are explicitly tested for 2.3.0 to 2.8.0)

    Warning: Installing on a CPU-only system will limit functionality as many CUDA-only operations will be unavailable.

  10. Understand Structured Point Clouds (SPC) representation

    master

    Structured Point Clouds (SPC) are a sparse octree-based representation used to organize and compress 3D geometrically sparse information (also known as sparse voxelgrids or quantized point clouds).

    Key characteristics:

    • Octree Structure: Uses a tree where each node has up to 8 children, representing a $(2, 2, 2)$ space partitioning.
    • Memory Layout: A batch of octrees is represented as a tensor of bytes (octrees). Each bit in the byte array represents binary occupancy in Morton Order (Z-order curve). Nodes follow a breadth-first-search order.
    • Feature Storage: Follows a Structure of Arrays (SoA) approach. Features (like RGB, normals, or activation maps) are stored as tensors of shape (num_nodes, feature_dim), where num_nodes is the number of nodes at a specific level.

    Warning: The SPC internal layout and structure is currently experimental and subject to change.

  11. Use the kaolin.physics module for representation-agnostic physics simulation

    master

    The kaolin.physics module provides tools for representation-agnostic physics simulation. It is organized into several submodules:

    • kaolin.physics.common: Common physics utilities.
    • kaolin.physics.materials: Physics material definitions.
    • kaolin.physics.simplicits: Core simulation engine components.
    • kaolin.physics.utils: Physics-related utility functions.

  12. Understand Newton-Simplicits Coupling Modes

    master

    Kaolin's Newton integration allows soft-body solvers (Simplicits) to interact with rigid bodies, robots, and MPM materials through two primary coupling patterns:

    1. Shared Model (Soft ↔ Rigid): Used when a single model instance is passed to multiple solvers (e.g., SimplicitsSolver and SolverSemiImplicit). Both solvers read and write to the same particle/body state arrays. SimplicitsSolver manages the Simplicits DOF range, while SolverSemiImplicit handles rigid bodies.

    2. Multiple Models (Particle ↔ Particle): Used for coupling different particle solvers (e.g., MPM and Simplicits). Each solver owns its own particle array, and they exchange forces via contact energy rather than shared state.