repository·main·Indexed 43 days ago
https://github.com/unique1i/scenesplatA Gaussian Splatting-based scene understanding framework utilizing vision-language pretraining for open-vocabulary 3DGS semantic learning and a generalizable 3DGS encoder. It supports self-supervised pretraining (MAE, SimDINO, iBOT), zero-shot semantic segmentation evaluation, and feature inference for 3DGS scenes.
Self-supervised pretraining uses Gaussian splat attributes. The dataset must follow this structure:
color.npycoord.npyopacity.npyquat.npyscale.npynormal.npy (optional, required for compatibility with PTv3 and Sonata)After downloading datasets (e.g., hypersim, arkitssene, 3rscan), update the data paths in your configuration file (e.g., configs/concat_dataset/ssl-pretrain-concat-scan-ppv2-matt-3rscan-arkit-hyper-mcmc-base.py).
To prepare Matterport3D 3DGS scenes, you must first run the preprocessing script and then the sampling/chunking script.
Note: Replace the placeholder paths with your actual local paths.
python -u preprocess_matterport3d_gs.py \
--pc_root /path/to/ptv3_preprocessed/matterport3d \
--gs_root /path/to/gaussian_world/matterport3d_region_mcmc_3dgs \
--output_root /path/to/gaussian_world/preprocessed/matterport3d_region_mcmc_3dgs \
--num_workers 8 \
--feat_root /path/to/gaussian_world/matterport3d_region_mcmc_3dgs/language_features_siglip2
python -u sampling_chunking_data_gs.py \
--dataset_root /path/to/gaussian_world/preprocessed/matterport3d_region_mcmc_3dgs \
--output_dir /path/to/gaussian_world/preprocessed/matterport3d_region_mcmc_3dgs \
--grid_size 0.01 --chunk_range 6 6 --chunk_stride 3 3 --split train --num_workers 12 --max_chunk_num 6 You can generate PCA visualizations of the features during or after inference.
During inference:
Add the --pca-vis flag to the lang_inference command. Use --no-save if you only want the visualizations and want to skip saving the large .pt feature tensor.
From existing feature files:
If you already have a .pt feature file, use scripts.pca_colorize_features.
Output files:
<scene>_pca_colored.ply: A colored point cloud using Gaussian centers.<scene>_feat_vis_3dgs.ply: A renderable 3DGS PLY that replaces SH color with PCA feature colors.# Option 1: During inference (with no intermediate .pt save)
python -m tools.lang_inference \
--config configs/inference/lang-pretrain-pt-v3m1-3dgs.py \
--checkpoint /path/to/model_best.pth \
--input-root /path/to/preprocessed_scene \
--output-dir /path/to/output_dir \
--pca-vis \
--no-save
# Option 2: From existing .pt file
python -m scripts.pca_colorize_features \
--feature-path /path/to/output_dir/<scene>_feat.pt \
--input-root /path/to/preprocessed_scene \
--output-dir /path/to/output_dirTo resume training, specify the path to your checkpoint using the weight option and set resume=True within the --options flag.
Note: If you provide weight but do not set resume=True, the training will load the weights but reset the epoch and scheduler, effectively starting a new training run from that state.
python tools/train.py \
--config-file lang-pretrain-concat-scan-ppv2-matt-mcmc-wo-normal-contrastive.py \
--options ...... \
weight=model_best.pth \
resume=True \
--num-gpus $gpu_num \To set up the SceneSplat environment, use the provided env.yaml file. The environment requires Python 3.10, PyTorch 2.5.1, and CUDA 12.4.
Run the following commands to create and activate the environment:
conda env create -f env.yaml
conda activate scene_splatCheckpoints from self-supervised pretraining must be converted before they can be used as weights for supervised semantic segmentation training. Use the scripts/rename_ckpt_dino_to_ptv3.py script for this conversion.
# 1. Convert checkpoint (conceptual step)
# python scripts/rename_ckpt_dino_to_ptv3.py <input_path> <output_path>
# 2. Use converted weight in supervised training
python tools/train.py \
--config-file configs/scannet/semseg-gs-scannet-all-w-fix-xyz.py \
--options save_path=exp_runs/scannet/<Experiment_Name> \
batch_size=$batch_size batch_size_val=$batch_size_val \
batch_size_test=$batch_size_test num_worker=$num_worker gpu_nums=$gpu_num \
weight=<model_extract_path> \
--num-gpus $gpu_num \Before running feature inference, you must convert a 3DGS .ply file into a preprocessed folder containing .npy arrays (e.g., coord.npy, color.npy, opacity.npy, scale.npy, and quat.npy). Use the scripts.preprocess_gs module to perform this conversion.
python -m scripts.preprocess_gs \
--input /path/to/scene.ply \
--output /path/to/preprocessed_scenePerform feature inference on a preprocessed 3DGS scene using a model checkpoint. The output is a feature tensor file <output-dir>/<scene>_feat.pt with shape (N, 768), where $N$ is the number of Gaussians. This tensor contains the full-resolution PT-v3 decoder output from the backbone module.
python -m tools.lang_inference \
--config configs/inference/lang-pretrain-pt-v3m1-3dgs.py \
--checkpoint /path/to/model_best.pth \
--input-root /path/to/preprocessed_scene \
--output-dir /path/to/output_dirTo start vision-language pretraining, use the tools/ssl_pretrain.py script. You can override configuration settings on-the-fly using the --options flag. The save_path option is required to specify where checkpoints, logs, and evaluation results will be stored.
python tools/ssl_pretrain.py \
--config-file configs/concat_dataset/ssl-pretrain-concat-scan-ppv2-matt-3rscan-arkit-hyper-mcmc-base.py \
--options \
save_path=exp_runs/ssl_pretrainer/ssl-pretrain-scannet-all-base \
batch_size=$batch_size batch_size_val=$batch_size_val \
batch_size_test=$batch_size_test num_worker=$num_worker gpu_nums=$gpu_num \
--num-gpus $gpu_num \The SceneSplat dataset is hosted on Hugging Face as SceneSplat-7K.
Note: You must read and accept the dataset terms here before gaining access. The dataset includes 3DGS scenes, frames metadata, and 2D language features used during the 3DGS language label collection process.
Multi-node training is supported on SLURM clusters using the NCCL library. To run a task across multiple nodes, use srun to launch the training script and include the --multi_node flag. You may need to configure cluster-specific environment variables like NCCL_SOCKET_IFNAME.
srun python tools/train.py \
...... \
--multi_nodeTo evaluate semantic segmentation on custom scenes, you must provide semantic labels and configure the tester.
1. Prepare Labels: Provide one of the following:
segment.npy: Per-gaussian labels (ensure shape matches other *.npy files).pc_segment.npy: Point cloud semantic segmentation labels from the original optimization scene (recommended).2. Encode Labels: Convert class labels into text embeddings using:
python -m scripts.encode_labels.py3. Configure Tester: In your tester configuration:
GenericGSDataset.class_names, text_embeddings, and excluded_classes.ignore_index is used.