DOCSEARCHbyAlgolia
Submit Docs

dn-splatter

repository·main·Indexed 21 days ago

https://github.com/maturk/dn-splatter

Implementations for depth and normal supervision of Gaussian Splatting models, including DN-Splatter and AGS-Mesh. The library provides tools for improving novel-view synthesis and mesh reconstruction using smartphone data, monocular normal map generation via Omnidata or DSINE, scale-aligned depth estimation, and mesh extraction using methods such as o3dtsdf and IsoOctree. It is compatible with nerfstudio and supports datasets like MuSHRoom, ScanNet++, and Replica.

Tokens
15.7K
Snippets
61
Records
73
Agent score
74%

What's inside dn-splatter

  1. Generate monocular normal maps using Omnidata or DSINE

    main

    To use monocular normal supervision, you must first generate normal maps using an external network. DN-Splatter supports Omnidata and DSINE.

    Omnidata Normals

    1. Download weights: python dn_splatter/data/download_scripts/download_omnidata.py
    2. Generate maps: Use python dn_splatter/scripts/normals_from_pretrain.py with the --resolution flag (use low for efficiency).

    DSINE Normals

    Generate maps using: python dn_splatter/scripts/normals_from_pretrain.py --data-dir [PATH_TO_DATA] --normal-format dsine.

    Note: Generated normals are saved to data_root/normals_from_pretrain. Omnidata normals are converted from OpenGL to OpenCV coordinates for consistency.

    # 1. Download Omnidata weights
python dn_splatter/data/download_scripts/download_omnidata.py

# 2. Generate Omnidata normals (low res recommended)
python dn_splatter/scripts/normals_from_pretrain.py \
        --data-dir [PATH_TO_DATA_ROOT] \
        --img-dir-name [DIR_NAME] \
        --resolution low

# 3. Generate DSINE normals
python dn_splatter/scripts/normals_from_pretrain.py --data-dir [PATH_TO_DATA] --normal-format dsine

  2. Extract a mesh from trained models

    main

    Use the gs-mesh command to extract meshes. The recommended method is o3dtsdf.

    General command:

    gs-mesh {dn, tsdf, o3dtsdf, sugar-coarse, gaussians, marching} --load-config [PATH] --output-dir [PATH]

    Supported exporters:

    gs-meshDescriptionRequires normals?
    dnBackproject depth and normal maps to PoissonYes
    tsdfTSDF Fusion algorithmNo
    o3dtsdfTSDF Fusion algorithm used in 2DGS paperNo
    sugar-coarseLevel set extractor from SuGaRBoth
    gaussiansUse Gaussian xyzs and normals to PoissonYes

    Specialized extraction: For smoother surfaces using IsoOctree:

    python dn_splatter/scripts/isooctree_dn.py <root_folder> --transformation_path <pose_json_path> \
--tsdf_rel 0.03 --output_mesh_file <output_path/output.ply> --subdivision_threshold=100

    Tip: For very small object captures, use tsdf with a 0.004 voxel size and 0.02 SDF trunc distance.

    # Recommended method
gs-mesh o3dtsdf --load-config [PATH] --output-dir [PATH]

  3. Install DN-Splatter via Conda and Pip

    main

    DN-Splatter is compatible with a nerfstudio conda environment. To install, activate your existing nerfstudio environment, clone the repository, and install the package in editable mode.

    Note: You must install setuptools==69.5.1 specifically to ensure compatibility.

    conda activate nerfstudio
git clone https://github.com/maturk/dn-splatter
cd dn_splatter/
pip install setuptools==69.5.1
pip install -e .

  4. Generate scale-aligned monocular depth estimates

    main

    If you have a COLMAP-processed dataset but no sensor depths, use align_depth.py to solve for scale ambiguity between monocular depths and COLMAP poses. This creates sfm_depths/ and mono_depth/ directories in your data root.

    Troubleshooting: If you encounter a TypeError regarding numpy.int64 in Torch, downgrade from Torch 2.1.2 to 2.0.1.

    python dn_splatter/scripts/align_depth.py --data [path_to_data_root] \
                                      --skip-colmap-to-depths, --no-skip-colmap-to-depths \
                                      --skip-mono-depth-creation, --no-skip-mono-depth-creation

  5. Evaluate DN-Splatter models

    main

    Evaluation can be performed on individual configs or across entire datasets.

    RGB, Depth, and Pointcloud Metrics:

    ns-eval --load-config [PATH_TO_CONFIG] --output-path [JSON_OUTPUT_PATH]

    Add --render-output-path [PATH_TO_IMAGES] to render train/eval images.

    Mesh Metrics (MuSHRoom):

    python dn_splatter/eval/eval_mesh_mushroom_vis_cull.py --gt_mesh_path [GT_Mesh_Path] --pred_mesh_path [Pred_Mesh_Path] --output --device [iphone/kinect]

    Mesh Metrics (ScanNet++/Replica or Custom):

    python dn_splatter/eval/eval_mesh_vis_cull.py --gt-mesh-path [GT_Mesh_Path] --pred-mesh-path [Pred_Mesh_Path] --transformation_file [Path_to_transform_file] --dataset_path [Dataset path]

    Note: --transformation_file should be the path to the transform.json generated during training (e.g., for ScanNet++).

    # Evaluate metrics
ns-eval --load-config [PATH_TO_CONFIG] --output-path [JSON_OUTPUT_PATH]

  6. Generate depth masks for AGS-Mesh

    main

    The AGS-Mesh method requires depth masks generated via a depth-to-normal consistency check. Use depth_normal_consistency.py to produce these masks.

    python dn_splatter/scripts/depth_normal_consistency.py --data-dir <dataset_path> --transforms_name <transforms.json> --normal-format [omnidata/dsine]

  7. Convert dataset to COLMAP format

    main

    If your dataset lacks camera pose information, use the following script to generate poses via COLMAP.

    python dn_splatter/scripts/convert_colmap.py --image-path [data_root/images] --use-gpu/--no-use-gpu

  8. Generate monocular depth without SfM alignment

    main

    To simply render monocular depths for a dataset without attempting scale alignment to SfM points, use the --skip_alignment flag.

    python dn_splatter/scripts/align_depth.py --data [path_to_data_root] --skip-colmap-to-depths --skip_alignment

  9. Install DN-Splatter using Pixi

    main

    Pixi can manage the installation of CUDA, PyTorch, and Nerfstudio for you. After installing the pixi package manager, clone the repo and run pixi install.

    Use pixi run example to run a built-in example or pixi shell to activate the environment.

    git clone https://github.com/maturk/dn-splatter
cd dn_splatter/
pixi install

# To run an example
pixi run example

# To activate environment
pixi shell

  10. Process custom RGB-D smartphone data

    main

    To use casually captured RGB-D streams (e.g., from iPhone/Android via SpectacularAI or sensor streams like Oak/RealSense/Kinect), follow these steps:

    1. Capture and Process: Use the process_sai.py script to convert inputs into a Nerfstudio-compatible format (calculating VIO poses and creating transforms.json).

      pip install spectacularAI
python dn_splatter/scripts/process_sai.py [PATH_TO_SAI_INPUT_FOLDER] [PATH_TO_OUTPUT_FOLDER]

    2. Train:

      ns-train dn-splatter --pipeline.model.use-depth-loss True \
                 --pipeline.model.depth-lambda 0.2 \
                 --pipeline.model.use-normal-loss True \
                 --pipeline.model.use-normal-tv-loss True \
                 --pipeline.model.normal-supervision depth \
                 normal-nerfstudio --data PATH_TO_DATA
    # Process input
python dn_splatter/scripts/process_sai.py [PATH_TO_SAI_INPUT_FOLDER] [PATH_TO_OUTPUT_FOLDER]

  11. Train DN-Splatter with SDFStudio Data

    main

    When using data downloaded via ns-download-data sdfstudio, use the gsdf dataparser.

    Important for gneusfacto: You must set the --load-for-sdfstudio True flag in gsdf (it defaults to False).

    For DN-Splatter specifically: Run with --pipeline.model.use-depth-loss False.

    # Training gneusfacto with SDFStudio data
ns-train gneusfacto --data ./datasets/DTU/scan65/ gsdf --load-for-sdfstudio True

# Training dn_splatter
ns-train dn_splatter --pipeline.model.use-depth-loss False --data ./datasets/DTU/scan65 gsdf

  12. Compute Mesh Evaluation Metrics

    main

    To evaluate mesh quality, use the visibility-culling scripts located in dn_splatter/eval/. These scripts cull the predicted mesh based on training camera view visibility, ensuring that regions of the mesh not seen in the training dataset are ignored during metric computation.

    Supported metrics:

    • accuracy (lower is better)
    • completeness (lower is better)
    • chamferL1 (lower is better)
    • normals_correctness (higher is better)
    • F-score (higher is better)
    # Standard mesh evaluation with visibility culling
python dn_splatter/eval/eval_mesh_vis_cull.py --path-to-pd-mesh [PATH_TO_PREDICTED_PLY] --path-to-gt-mesh [PATH_TO_GT_PLY]

# For the MuSHRoom dataset specifically
python dn_splatter/eval/eval_mesh_mushroom_vis_cull.py --path-to-pd-mesh [PATH_TO_PREDICTED_PLY] --path-to-gt-mesh [PATH_TO_GT_PLY]