DOCSEARCHbyAlgolia
Submit Docs

3DGRUT Documentation

repository·main·Indexed 25 days ago

https://github.com/nv-tlabs/3dgrut

Official implementation of 3D Gaussian Ray Tracing (3DGRT) and 3D Gaussian Unscented Transform (3DGUT). The library enables high-quality rendering of volumetric Gaussian particles and supports complex camera effects such as rolling shutters and secondary rays (reflection, refraction, shadows) using a hybrid approach of rasterization and ray tracing. It includes tools for training on benchmark datasets (NeRF Synthetic, MipNeRF360, ScanNet++), exporting scenes to USD, PLY, or NuRec, and an interactive playground for scene exploration.

Tokens
15.9K
Snippets
33
Records
71
Agent score
82%

What's inside 3DGRUT

  1. Configure Mesh Primitive Types

    main

    The Playground allows adding, removing, and duplicating geometries via the Primitives subsection. The default geometries include a Quad and a Sphere, plus any additional .mesh files located in threedgrut_playground/assets.

    When adding a primitive, you can change its Type to modify how rays interact with the mesh:

    • Glass: Follows Snell's law to refract rays. You can adjust the Refractive Index (IOR) to change the perceived thickness of the medium.
    • Mirror: Acts as a perfect mirror, reflecting rays around the mesh normal.
    • Diffused Mesh: Uses Lambertian shading for diffusive materials. These can be assigned different materials from the Materials section.
    • PBR Mesh: Uses Cook-Torrance shading for Physically Based materials. This mode supports BRDF and BTDF and requires antialiasing and higher Samples per Pixel (SPP) for high-quality rendering.

  2. How PPISP integration modes work: spg-runtime vs sh-optimized

    main

    PPISP (Physically-Plausible Image Signal Processing) can be integrated into USD assets in two ways depending on your target viewer's capabilities:

    1. spg-runtime (Default)

      • Mechanism: PPISP is authored as Omniverse Sensor Processing Graph (SPG) shader prims on every RenderProduct.
      • Behavior: The viewer applies vignetting, ZCA color, exposure, and CRF at render time.
      • Pros/Cons: Highest fidelity, but requires an SPG-capable viewer (e.g., Omniverse Kit / RTX).
      • Best for: High-end Omniverse pipelines.

    2. sh-optimized

      • Mechanism: PPISP is folded (baked) into the Gaussian spherical harmonic (SH) coefficients during export.
      • Behavior: The rendered image already looks PPISP-corrected without needing runtime shaders.
      • Pros/Cons: Lower fidelity (it is a per-asset baked approximation tied to a single reference camera/frame), but works in any USD viewer.
      • Best for: General USD compatibility where runtime PPISP is unavailable.

  3. How 3DGRT hybrid rendering works

    main

    The Playground uses a hybrid rendering approach that combines Gaussian particles and surface mesh primitives via ray tracing. The rendering loop for each frame follows these steps:

    1. Ray Tracing: A ray is traced from the origin against a BVH (Bounding Volume Hierarchy) containing all mesh primitives. If the ray hits a surface, the hit point $\mathbf{x}$ is identified, and the ray may be redirected or shaded based on the mesh properties.
    2. Volumetric Integration: A 3DGRT volumetric integration phase runs for the segment between the ray origin $\mathbf{r}_o$ and the hit point $\mathbf{x}$ to accumulate radiance from the Gaussians.
    3. Mesh Contribution: If the mesh surface contributed radiance, it is factored into the total.
    4. Iteration: The process loops, using the hit point $\mathbf{x}$ as the new ray origin for the next trace (following the redirection).

    This continues until the ray misses or sufficient radiance is accumulated. If a ray misses, environment maps (.hdr) can contribute radiance to light PBR (Physically Based Rendering) primitives.

  4. Configure Materials and Environment Maps

    main

    Materials

    The Materials section provides a property editor for all loaded materials. By default, solid and checkboard materials are available. If .gltf or .glb files are loaded, their associated materials will also appear here.

    Environment Maps

    Environment maps provide global lighting (IBL) and override the background. Note that env maps are only enabled when path tracing is active and at least one mesh primitive is present.

    Key controls for Environment Maps:

    • IBL Intensity: A linear scalar multiplier applied to the env map before path tracing.
    • Exposure: Scales the HDR range using the formula $hdr = hdr * 2^{exposure}$ before tone mapping.
    • Offset Sliders: Adjust the position of the environment map.
    • Tone Mapping & Gamma Correction: Applied after exposure to finalize the image.

  5. Use static SPG fallback for PPISP

    main

    When a controller is not present or is explicitly disabled via --disable-ppisp-controller-export, the exporter uses a static spg-runtime path. The authoring strategy depends on the reference camera and frame IDs provided:

    --ppisp-reference-camera-id--ppisp-reference-frame-idAuthored Strategy
    unsetunsetTime-sampled per-camera (one USD time sample per training frame)
    setunsetStatic, single-camera, neutral exposure and identity color latents
    setsetStatic, single-frame: exposure/color latents from frame_id
    unsetsetREJECTED (Frame-only fixing is ambiguous for vignetting/CRF)

    Note: Setting a frame ID without a camera ID will trigger a rejection error because vignetting and CRF (Camera Response Function) are anchored to the camera axis.

  6. How PPISP USD Export works with Omniverse Kit

    main
    When exporting to Omniverse Kit, the exporter creates a hidden <cam>_ppisp sibling camera for the rig camera. This sibling camera inherits the rig camera's intrinsics but neutralizes the standard exposure:* namespace (setting exposure = 0.0, exposure:fStop = 1.0, etc.) to prevent the double-exposure effect that occurs when both Kit's exposure model and the SPG PPISP shader are applied. The RenderProduct's camera relationship is rewired to this hidden camera, which carries the PPISP parameters as ppisp:* attributes.

  7. Understand the Authored USD Surface (Standard vs Omniverse)

    main

    The PPISP export produces both standard USD primitives and Omniverse/RTX-specific extensions.

    Standard USD (Universal)

    • UsdVol ParticleField3DGaussianSplat (or UsdGeomPoints): The Gaussian payload.
    • UsdLux DomeLight: The trained sky environment.
    • Camera: The original rig cameras (untouched).

    Omniverse / RTX-specific (SPG-capable viewers)

    These are conventions consumed by Omniverse's RTX renderer and SPG runtime:

    • SPG Sidecar Files:
      • Static: ppisp_usd_spg.cu, ppisp_usd_spg.cu.lua, ppisp_usd_spg.usda.
      • Controller: ppisp_controller_<camera>.cu, ppisp_controller_<camera>.cu.lua, ppisp_usd_spg_auto.*.
    • PPISP Shader Prims: Authored on every RenderProduct. They connect HdrColor to PPISPColor and rewire the display LdrColor to PPISPColor.
    • <cam>_ppisp Camera: A hidden sibling of the rig camera that carries all ppisp:* attributes (exposure, color, etc.). The RenderProduct is rewired to this hidden camera.
    • Controller Prims (Controller path only): PPISPControllerPool and PPISPController shader prims wired to the ControllerParams AOV.

  8. Use SH-optimized export mode

    main

    The sh-optimized mode folds PPISP into Spherical Harmonics (SH). This mode is intended for non-SPG viewers.

    Important constraints:

    • --ppisp-responsivity is a no-op in this mode; it is not used because no PPISP shader is authored. Use --scene-radiance-scale as your primary knob for radiance matching.
    • --enable-ppisp-controller-export is incompatible with sh-optimized and will be rejected by the validator.
    • The SH bake runs before radiance scaling is applied to ensure the optimization process is not undone by the scaling.

  9. Use Optix Denoiser and Depth of Field

    main

    Optix Denoiser

    To reduce noise in rendered images, enable the learned denoiser via Render > Use Optix Denoiser. This can be used independently or in conjunction with antialiasing.

    Depth of Field

    To simulate a lens effect that blurs areas outside the focus region, use the Depth of Field settings. This effect applies to both 3D Gaussians and mesh primitives. Controls include:

    • Samples per Pixel (SPP)
    • Aperture Size
    • Focus Z

  10. Supported 3DGRUT Export Formats

    main

    3DGRUT supports three primary export formats for Gaussian models:

    • PLY: Point cloud format containing pre-activation data. Best for interchange with standard 3DGS tooling.
    • USD (standard / lightfield): Uses the UsdVol ParticleField3DGaussianSplat schema. This is the native USD Gaussian asset format for USD-aware renderers. Note: Prefer this over NuRec for new assets.
    • NuRec: A NuRec Volume USDZ format. Compatible with Omniverse Kit 107.3 - 110.1 and Isaac Sim 5.0 - 6.0. This format is slated for deprecation in favor of ParticleField.

  11. Convert and re-export Gaussian assets

    main

    The repository includes tools for asset preparation, allowing you to convert, combine, partition, and re-export Gaussian assets between the following formats:

    • PLY
    • ParticleField USD
    • NuRec

    Detailed instructions for transcoding between these formats can be found in the threedgrut/export/README.md file.

  12. Run the Interactive Playground GUI

    main

    The playground allows for interactive exploration of pretrained scenes, supporting ray-tracing effects like reflections, refractions, depth of field, and inserted objects.

    To visualize a pretrained scene, run the playground.py script and provide the path to the Gaussian Splat (gs_object) checkpoint.

    python playground.py --gs_object <ckpt_path>