DOCSEARCHbyAlgolia
Submit Docs

Sparse Voxels Rasterizer (svraster)

repository·main·Indexed 21 days ago

https://github.com/nvlabs/svraster

An efficient radiance field rendering algorithm that utilizes adaptive sparse voxels instead of neural networks or 3D Gaussians to achieve high frame rates and resolution. The library includes tools for scene optimization, mesh extraction, and an interactive viewer. It supports various initialization methods for SparseVoxelModel (dense grid, voxel indices, or point clouds) and provides an evaluation toolbox for the TanksAndTemples benchmark.

Tokens
13.6K
Snippets
48
Records
62
Agent score
75%

What's inside svraster

  1. Access model properties and derived attributes

    main

    The model instance provides several properties to inspect the scene structure and voxel data:

    Scene and Voxel Counts

    • num_voxels: Total number of voxels.
    • num_grid_pts: Total number of unique grid points (a voxel has 8 corner grid points, but points are shared between adjacent voxels).

    Spatial Bounds

    • scene_min / scene_max: The minimum and maximum coordinates of the entire scene.
    • inside_min / inside_max: The minimum and maximum coordinates of the main foreground region (valid if the model was created with outside_level > 0).
    • inside_mask: A mask indicating which voxels fall within the inside_min and inside_max bounds.

    Subdivision Tracking

    • subdivision_priority: Tracks and accumulates subdivision priority during the rendering backward pass. Higher values indicate higher priority. Use reset_subdivision_priority() to clear this.

    Lazily Computed Properties

    These are computed upon first access and automatically recompute when voxel allocation changes (e.g., after pruning or subdivision):

    • vox_center: Voxel center positions in world space.
    • vox_size: Size of the voxels.
    • vox_key: Index to unique grid points, shaped [num_voxels, 8].
    • grid_pts_xyz: World-space positions of the unique grid points.

  2. Understand the Gaussian-Splatting License terms

    main

    The software is distributed under a specific license held by Inria and the Max Planck Institut für Informatik (MPII).

    Usage Rights

    • Research Use: Non-exclusive, royalty-free rights are granted to academic and industrial research users for research and/or evaluation purposes.
    • Derivative Works: You may reproduce, prepare derivative works of, publicly display, publicly perform, and distribute your work, provided you follow the redistribution rules.

    Key Limitations

    • Non-Commercial Restriction: The software may be used only for research and/or evaluation purposes. Commercial use, exploitation, or distribution is strictly prohibited without prior and explicit consent from the licensors.
    • Redistribution Requirements: If you redistribute the work, you must:
      1. Use this same License.
      2. Include a complete copy of this License.
      3. Retain all original copyright, patent, trademark, or attribution notices.
    • Derivative Terms: If you apply different terms to your derivative works ("Your Terms"), those terms must still respect the use limitations defined in this license.

    Commercial Inquiries

    For any unauthorized or commercial use, you must contact Inria at: stip-sophia.transfert@inria.fr.

  3. Install Sparse Voxels Rasterizer

    main

    Follow these steps to set up the environment. Ensure you have PyTorch installed first (tested with 1.13.1+cu117 and 2.5.0+cu124).

    1. Install the appropriate cuda-toolkit for your PyTorch version using Conda:
      • For CUDA 11.7: conda install -y -c "nvidia/label/cuda-11.7.0" cuda-toolkit
      • For CUDA 12.4: conda install -y -c "nvidia/label/cuda-12.4.0" cuda-toolkit
    2. Install Python dependencies: pip install -r requirements.txt
    3. Install the sparse voxel CUDA rasterizer and utilities in editable mode: pip install -e cuda/
    conda install -y -c "nvidia/label/cuda-11.7.0" cuda-toolkit
conda install -y -c "nvidia/label/cuda-12.4.0" cuda-toolkit
pip install -r requirements.txt
pip install -e cuda/

  4. Run the TanksAndTemples evaluation

    main

    To evaluate a specific scene (e.g., Barn), follow these steps:

    1. Cull the mesh: Run cull_mesh.py first to prepare the mesh.
    2. Execute evaluation: Run the provided shell script ./run.sh with the scene name as an argument.

    Example for the Barn scene:

    ./run.sh Barn
    # Step 1: Cull mesh (assumed command)
python cull_mesh.py

# Step 2: Run evaluation
./run.sh Barn

  5. Render views and measure FPS

    main

    Use render.py to generate images and evaluate performance.

    TaskCommand
    Measure FPSpython render.py $OUTPUT_PATH --eval_fps
    Full training viewspython render.py $OUTPUT_PATH --skip_test --rgb_only --use_jpg
    Testing viewspython render.py $OUTPUT_PATH --skip_train (requires --eval during training)
    Fly-through videopython render_fly_through.py $OUTPUT_PATH
    python render.py $OUTPUT_PATH --eval_fps

  6. Extract a mesh from the voxel field

    main

    To generate a mesh, it is recommended to train with normal consistency losses (--lambda_normal_dmean 0.001 --lambda_normal_dmed 0.001) and potentially sparse depth guidance (--lambda_sparse_depth 0.01).

    Once optimization is complete, run:

    python extract_mesh.py $OUTPUT_PATH
    python extract_mesh.py $OUTPUT_PATH

  7. Use the interactive viewer

    main

    To navigate a trained scene in a web browser, run:

    python viz.py $OUTPUT_PATH

    Note: The FPS of the visualizer may be limited by network streaming if running on a remote server. WebGL support is available via the svraster-webgl viewer.

    python viz.py $OUTPUT_PATH

  8. Prepare ScanNet++ dataset for SVR

    main

    To use the ScanNet++ dataset with Sparse Voxels Rasterizer (SVR), follow these steps:

    1. Download the source data from the ScanNet++ official site.
    2. Run the preprocessing script to convert the source data into the format required by SVR.

    Use the scripts/scannetpp_preproc.py script with the following arguments:

    • --indir: Path to your downloaded source data.
    • --outdir: Destination directory for processed data (e.g., data/scannetpp_nvs).
    • --ids: The sequence of scene IDs you wish to process.
    python scripts/scannetpp_preproc.py --indir $PATH_TO_SOURCE_DATA --outdir data/scannetpp_nvs --ids $SEQUENCE_OF_SCENE_ID

  9. Prepare data for reconstruction

    main

    To reconstruct a scene from captures, you must first extract camera parameters using COLMAP (via InstantNGP or NerfStudio).

    Requirement: The system currently only supports pinhole camera mode. Ensure your preprocessing uses:

    • InstantNGP: --colmap_camera_model PINHOLE
    • NerfStudio: --camera-type pinhole

  10. Optimize a scene

    main

    Run the training script to optimize the sparse voxel field. All outputs (config, visualizations, statistics) are saved to the specified $OUTPUT_PATH.

    python train.py --eval --source_path $DATA_PATH --model_path $OUTPUT_PATH

    Configuration Hierarchy

    Configuration is applied in this order (later values overwrite earlier ones):

    1. src/config.py: Default values.
    2. --cfg_files: List of config files (e.g., from cfg/).
    3. Command line arguments: Any field in src/config.py can be overwritten directly (e.g., --data_device cpu).
    python train.py --eval --source_path $DATA_PATH --model_path $OUTPUT_PATH

  11. Set up the TanksAndTemples evaluation toolbox

    main

    The evaluation toolbox is a Python-based tool used to evaluate training datasets from the TanksAndTemples benchmark. It requires reconstructing 3D models and camera poses from raw videos before running the evaluation.

    Prerequisites

    1. Reconstruct Data: Reconstruct 3D models and recover camera poses from the raw TanksAndTemples training videos (available at tanksandtemples.org).
    2. Download Ground Truth: Download the evaluation data (ground truth geometry + reference reconstruction) from this Google Drive link. Assume the dataset folder is TanksAndTemples/evaluation/data/.

    Installation

    Ensure you have Python 3 installed and install the following dependencies:

    • open3d (specifically version v0.9.0 is mentioned)
    • matplotlib
    • numpy
    • json
    pip install open3d==0.9.0 matplotlib numpy

  12. Reduce model size using quantization in SVInOut

    main

    When saving models via SVInOut.save(path, quantize=True), the model uses a quantization scheme to reduce storage requirements. This process compresses _geo_grid_pts, _sh0, and _shs tensors.

    Benefits:

    • Reduces model size by approximately 70%.
    • Results in a minor drop in PSNR.

    Mechanism: It uses a codebook-based quantization where values are mapped to indices. The quantize flag in save and the automatic detection of the quantized key in load ensure that the model is correctly compressed and decompressed during I/O operations.