Unity Gaussian Splatting

To combine multiple splat clouds into a single entity, select multiple GaussianSplatRenderer objects in the hierarchy. A Merge button will appear in the inspector.

Clicking this will:

1. Merge the selected splats into one single renderer.
2. Disable the GameObjects of the original redundant renderers.

The resulting merged splat can then be edited manually or exported to the PLY format.

The GaussianCutout component allows you to "virtually delete" splats within a specific volume. Cutouts can be shaped as an Ellipsoid or a Box, with their volume defined by the object's Transform (position, rotation, and scale).

Visibility Logic:

• Default mode: Hides all splats outside the cutout. If multiple cutouts exist, a splat is visible if it is inside any cutout.
• Invert mode: Enable the Invert toggle to hide splats inside the cutout. If multiple cutouts exist, a splat is hidden if it is inside any inverted cutout.

Cutouts are managed via the GaussianSplatRenderer inspector, which maintains a list of cutout objects affecting the splat cloud.

The project requires DX12 or Vulkan on Windows. DX11 is not supported.

To fix this, go to your Player Settings -> Graphics APIs section and ensure Windows is set to use DX12.

To convert raw Gaussian Splat data into Unity-compatible assets:

1. In the Unity Editor, navigate to the menu: Tools -> Gaussian Splats -> Create GaussianSplatAsset.
2. In the dialog, set the Input PLY/SPZ File to your source file.
3. Supported Formats:
   • PLY format: From the original 3DGS paper (typically found in point_cloud/iteration_*/point_cloud.ply).
   • Scaniverse SPZ format.
4. (Optional) Provide a cameras.json file in the same directory or a parent folder.
5. Select your desired compression options and output folder.
6. Click Create Asset.

Successful creation will result in a GaussianSplat asset accompanied by several data files.

To display a Gaussian Splat in your scene:

1. Create or locate a GameObject that has the GaussianSplatRenderer script attached.
2. In the Inspector, assign your created GaussianSplat asset to the Asset field.

Notes:

• The renderer respects the GameObject's transformation matrix. (Note: Official paper models often require a specific rotation/mirroring transform to align correctly).
• The script includes controls for debugging/visualizing data and a slider to move the game camera to specific asset camera locations.

To use this project as a starting point, follow these steps:

1. Download or clone the repository.
2. Open the projects/GaussianExample folder as a Unity project (Unity 2022.3 is recommended).
3. Open the GSTestScene scene.

Platform Requirements:

• Windows: Requires DX12 or Vulkan. DX11 is not supported.
• Mac: Works with Metal.
• Linux: Works with Vulkan.
• VR: Known to work on HTC Vive, Varjo Aero, Quest 3, and Quest Pro. (Apple Vision Pro is not confirmed).
• Note: Mobile and Web support is not tested and likely does not work.

The project supports Built-in, URP, and HDRP. Follow the specific setup instructions for your pipeline:

• Built-in Render Pipeline (BiRP): No extra setup required. Just add GaussianSplatRenderer components to your objects.
• Universal Render Pipeline (URP): Add GaussianSplatURPFeature to your URP renderer settings.
  • Requirements: Unity 6 or later.
  • Constraint: Render Graph "Compatibility Mode" in URP settings must be turned off.
• High Definition Render Pipeline (HDRP): Add a CustomPass volume object and include a GaussianSplatHDRPPass entry.
  • Tip: Rendering after postprocess often produces better results than rendering before transparencies, as it avoids issues with HDRP auto-exposure.

You can manually select and manipulate splats using the GaussianSplatRenderer inspector or the splat edit tool (the "blob" icon) in the scene view toolbar.

Important: There is no Undo for these operations. To revert edits, disable and then re-enable the GaussianSplatRenderer component.

Selection and Manipulation Shortcuts:

• Click-drag: Select splats within a rectangle.
• Shift-drag: Add to current selection.
• Ctrl-drag: Remove from selection.
• Backspace / Delete: Delete selected splats.
• Ctrl/Cmd+I: Invert selection.
• Ctrl/Cmd+A: Select all splats.
• F: Frame view on selected splats.
• W (Move tool): Move selected splats in space.

Best Practice: Use the "Very High" quality preset when importing the original Gaussian Splat file to avoid editing quantized or lossy representations.

After performing manual edits or creating cutouts, you can export the modified result back into a standard Gaussian Splat .ply file using the Export modified PLY button in the GaussianSplatRenderer inspector. This file can be used in other applications or re-imported into this project.

Export Options:

• World Space Export: When enabled, the splat's Transform (position, rotation, and scale) will be "baked" into the exported PLY file, including the proper rotation of spherical harmonics.

If you are using URP and notice rendering issues, check the following:

• Upside-down rendering: This occurs if you turn off HDR and change the Intermediate Texture setting from the default "Always" to "Auto".
• Compatibility Mode: Ensure Render Graph "Compatibility Mode" is disabled in URP settings.

Multiple GaussianSplatRenderer objects are roughly ordered by their Transform positions. If objects overlap significantly, you may see incorrect rendering results.

You can manually control the rendering priority using the Render Order setting on the splat object:

• Objects with a higher Render Order are always rendered "on top" (in front) of objects with a lower order.
• Objects within the same Render Order are sorted by their relative distance to the camera.

When integrating Gaussian Splats into a scene, be aware of the following rendering behaviors:

• Occlusion: Splats are rendered after opaque objects and the skybox and are tested against the Z buffer. This means opaque objects can occlude splats correctly.
• Transparency Interaction: Splats do not write to the Z buffer and are rendered before all other transparencies. They will not interact correctly with semitransparent objects.
• Lighting: Splats are not affected by lighting, shadows, reflection probes, lightmaps, or the skybox.
• Anti-aliasing: MSAA (Multi-Sample Anti-Aliasing) is not supported and will not work with Gaussian Splatting.