DOCSEARCHbyAlgolia
Submit Docs

SPAG-4D: 360° Panorama to 3D Gaussian Splat Converter

repository·main·Indexed 42 days ago

https://github.com/cedarconnor/spag4d

A tool for converting 360° equirectangular panoramic photos into 3D Gaussian Splat scenes. Version 3.0.0 features multiple generator backends including DA360, DAP, PaGeR, and SHARP 360, as well as refinement backends like OmniRoam v2, Geometric, and ArtiFixer3D to fill structural holes. Includes a Python API and CLI for transforming panorama images into .ply files, with optional neural upscaling via SeedVR2.

Tokens
100.3K
Snippets
285
Records
390
Agent score
53%

What's inside SPAG-4D

  1. Understand the SPAG-4D project structure

    main

    The project is organized into two main functional areas:

    spag4d/ (Core Conversion Pipeline)

    Contains the orchestrator (core.py) and various generator implementations:

    • Depth-based generators: da360_model.py (default), dap_model.py, and pager_model.py.
    • ML-based generators: sharp360.py and unisharp360.py (via unisharp_adapter.py).
    • Utilities: spag_converter.py (Depth $\rightarrow$ Gaussian projection), ply_writer.py (PLY export), spherical_grid.py (coordinate math), and cli.py (CLI interface).
    • Adapters: seedvr2.py for Windows SeedVR2 upscaling.

    spag4d/refine/ (Refinement Pipeline)

    Handles post-processing and distillation:

    • pipeline_v2.py: OmniRoam v2 distillation.
    • geometric/: A multi-stage geometric refinement pipeline (depth-align $\rightarrow$ hole-filter $\rightarrow$ consistency $\rightarrow$ aggregate $\rightarrow$ inject).
    • Adapters: omniroam_adapter.py (WSL2 wrapper) and camera_rig.py (novel-view camera extraction).

    Other Directories

    • api.py: FastAPI server providing /api/convert and /api/refine_v2 endpoints.
    • static/: Web UI assets.
    • scripts/: Setup scripts (e.g., setup_omniroam_wsl.sh).

  2. What is PaGeR and how does it integrate with SPAG-4D?

    main

    PaGeR ("Unified Panoramic Geometry Estimation via Multi-View Foundation Models") is a panoramic geometry model that acts as a fourth depth/geometry backend in SPAG-4D, alongside DA360, DAP, and SHARP360.

    Unlike previous depth estimators that only provide depth, PaGeR is a single-forward-pass model that outputs:

    1. Scale-invariant (SI) depth
    2. Metric depth
    3. World-frame surface normals
    4. A learned sky mask

    In the SPAG-4D pipeline, PaGeR is a drop-in generator. Its depth output is used for scene conversion, while its sky mask and surface normals are used as additive, optional signals to improve downstream stages: the sky mask replaces percentile-based sky heuristics to exclude sky from depth-range fitting, and the normals provide a direct signal for grazing-angle/floater pruning.

  3. Configure camera modes for UniSHARP

    main

    When running UniSHARP inference, you can specify the camera model using the --camera flag.

    For 2:1 Equirectangular (ERP) panoramas, use --camera panorama or --camera erp. This enables native ERP inference and skips the cubemap decomposition process.

    Camera Resolution Precedence:

    1. Explicit --camera panorama|erp
    2. Explicit --camera perspective|pinhole|fisheye
    3. camera field within a --camera-json file
    4. Aspect-ratio auto-detection (triggered if aspect ratio is between 1.9 and 2.1)
    # Example for a native panorama
python scripts/infer_unisharp.py --checkpoint step_123.pt --image pano.jpg --camera panorama --save-ply

  4. Compare SPAG and SHARP pipeline modes

    main

    SPAG-4D offers two primary processing modes:

    SPAG Mode (Default)

    Fast depth-to-Gaussian conversion using spherical projection. Colors are sampled directly from panorama pixels. Quality is controlled via the Stride parameter:

    • Stride 1: ~1.3M splats (Maximum quality)
    • Stride 2: ~350K splats (Default balance)
    • Stride 4: ~85K splats (Fast preview)
    • Stride 8: ~21K splats (Ultra-fast)

    SHARP Refined Mode

    Enabled via the SHARP Refine checkbox in the UI or the --sharp-refine flag in the CLI. It uses per-face neural Gaussian prediction. This mode is significantly slower (~60s+) but produces much higher detail per face.

  5. Understand PLY provenance metadata for geometric refinement

    main

    SPAG-4D extends the standard 3DGS PLY format with a custom attribute block to track the provenance of Gaussians. This allows the system to distinguish between base panorama Gaussians and those added during the geometric refinement process.

    When loading a PLY, you can use the load_provenance=True keyword argument in the SPAG-4D reader to populate a GaussianProvenance structure on the GaussianModel. A diagnostics.json sidecar file may also be generated containing per-voxel contributor lists.

    Custom PLY Attributes:

    • source_kind: (uchar) 0=base_panorama, 1=omniroam_geometric, etc.
    • primary_source_frame_idx: (int) -1 for base.
    • support_count: (uchar) 0 for base.
    • alignment_scale: (float) NaN for base.
    • alignment_shift: (float) NaN for base.
    • hole_mode: (uchar) 0=none, 1=alpha, 2=disoccl.
    • positional_std: (float) 0.0 for base.

  6. Understand the UniSHARP integration strategy for SPAG4d

    main

    UniSHARP is integrated into SPAG4d as a native-ERP quality upgrade rather than a way to increase camera travel distance. Its novel-view baseline is small (approximately 0.2m forward / 0.1m rotation radius).

    In the SPAG4d pipeline, UniSHARP serves two primary high-value roles:

    1. Cleaner native-ERP geometry: Providing geometry without the seams typically found in cubemap-based SHARP results.
    2. Hybrid pole/seam filler: Acting as a filler for the nadir/zenith/pole areas and seams when combined with the existing SHARP pipeline.

    When building or using hybrid modes, focus on leveraging these strengths rather than expecting a larger reconstruction envelope.

  7. Review SPAG-4D licensing and commercial usage

    main

    SPAG-4D uses a mix of licenses depending on the backend component used. Use the following guide to determine if your use case is commercially viable:

    ComponentLicenseCommercial Use
    SPAG-4D coreMITOK
    DA360 / DAPMITOK
    SeedVR2MITOK
    SHARP (Apple)Apple model licenseNon-commercial research only
    PaGeR weightsCC BY-NC 4.0Non-commercial / evaluation only
    OmniRoamAdobe Research LicenseNon-commercial research only

  8. Compare SPAG-4D Generators

    main

    SPAG-4D provides four main generator backends for converting 360° equirectangular panoramas into 3D Gaussian Splats. Choose based on your VRAM, speed, and licensing needs:

    GeneratorMethodSpeedQualityVRAMLicense
    DA360 (default)Depth + Spherical Projection~2 sGood, seamless poles~2 GBCommercial-OK
    DAPMetric Depth + Projection~3 sGood, metric scale~3 GBCommercial-OK
    PaGeRCubemap depth + learned sky/normals~15 sStrongest outdoor depth~12 GBNon-commercial
    SHARP 360ML-based per-face prediction~30 sHighest per-face detail~8 GBNon-commercial

    Note on Licensing: DA360 and DAP are commercial-safe. PaGeR and SHARP 360 use research weights with non-commercial licenses.

  9. Use the SHARP 360 generator for splat creation

    main

    The SHARP 360 generator uses Apple's SHARP model to predict Gaussians from perspective faces extracted from an equirectangular panorama. It includes an optional SeedVR2 upscaling step and uses DA360 for inter-face alignment to ensure geometric consistency.

    Pipeline Workflow:

    1. Panorama Loading: Validates 2:1 equirectangular format.
    2. Extraction Layout: Computes $N$ horizon views (default 6) with overlap.
    3. Perspective Extraction: Converts ERP to perspective views.
    4. SeedVR2 Upscale (Optional): Upscales faces using native Windows SeedVR2.
    5. SHARP Prediction: Runs ML inference per face to produce Gaussians3D.
    6. Border Clipping: Uses hard Voronoi clipping to prevent ghosting at seams.
    7. DA360 Alignment: Uses a DA360 disparity map to align SHARP depths via a smooth grid scale field.
    8. Merge & Export: Rotates Gaussians to the world frame and exports to a standard PLY file.
    # Example CLI usage for SHARP 360
python -m spag4d convert panorama.jpg output.ply --generator sharp360 --side-count 6 --seedvr2-upscale

  10. Integrate PaGeR as a depth/geometry backend

    main

    PaGeR (Panoramic Geometry Reconstruction) can be integrated into the SPAG-4D pipeline as a fourth depth/geometry generator, alongside DA360 and DAP. It is designed as a drop-in addition that follows the existing model wrapper contract: it returns a depth array that spag_converter.py uses to project Gaussians.

    Beyond depth, PaGeR provides two optional additive outputs that can enhance downstream stages:

    • Sky Mask (sky_mask): A boolean (H,W) array that can replace the existing percentile-based sky cutoff heuristic in scene_analysis.py.
    • Surface Normals (normals): An (H,W,3) array that can be used by scene_filter.py for grazing-angle and outlier pruning.

    Key Integration Note: To avoid geometry softening at depth discontinuities (due to PaGeR's resolution limits compared to SPAG-4D's high-res color sources), you should upsample the depth map to your working resolution before projection.

    360 ERP panorama
  ├─ DA360 ───────────────────┐
  ├─ DAP   ───────────────────┤
  ├─ SHARP360 ────────────────┤
  └─ PaGeR (pager_model.py) ──┘→ depth (H,W) ──→ scene_analysis.py ──→ spag_converter.py ──→ scene_filter.py ──→ ply_writer.py
       │                                              ▲                                          ▲
       ├─ sky_mask (H,W) bool ────────────────────────┘ (replaces percentile sky cutoff)        │
       └─ normals (H,W,3) ──────────────────────────────────────────────────────────────────────┘ (feeds grazing-angle + outlier pruning)

  11. Understand the GSFix3D Refine Module Architecture

    main

    The GSFix3D Refine module replaces the legacy Klein-based refinement pipeline with a disocclusion repair system using differentiable rendering. The refinement process follows a three-phase architecture:

    1. Camera Rig Generation & Hole Detection: Generates a camera rig and uses novel-view rendering to identify holes in the scene via mask extraction.
    2. GSFixer Diffusion Model: Performs fine-tuning and inference using the GSFixer diffusion model to repair identified holes.
    3. 3DGS Distillation: Optimizes the 3D Gaussian Splats (3DGS) against the repaired images using a distillation loop (L1 + D-SSIM loss and densification).

    Technical Requirements:

    • Python 3.10+
    • PyTorch 2.4
    • diff-gaussian-rasterization (CUDA extension)
    • Stable Diffusion v2 (via MarigoldGSFixerPipeline)
    • Dependencies: trimesh, open3d, FastAPI, GaussianSplats3D viewer.

  12. How PaGeR integration improves scene analysis and filtering

    main

    Integrating PaGeR into the SPAG-4D pipeline provides two major improvements over traditional depth-only methods:

    1. Robust Scene Analysis: When a sky_mask is provided by PaGeR, analyze_scene() skips percentile-based sky detection. It computes depth_min, depth_max, and orbit_radius using only non-sky pixels. This prevents sky pixels (which often read as maximum depth) from skewing the auto depth-range fit, making outdoor captures much more robust.

    2. Normal-Aware Pruning: Instead of inferring surface orientation from noisy depth gradients, the pipeline can use PaGeR's explicit normals.

      • Grazing-angle clip: Uses dot(view_ray, normal) to clip Gaussians, providing a cleaner signal than depth-gradient estimation.
      • Outlier/Floater pruning: Uses normal consistency among neighbors as a secondary signal to identify floaters.