DOCSEARCHbyAlgolia
Submit Docs

DreamGaussian Documentation

repository·main·Indexed 26 days ago

https://github.com/dreamgaussian/dreamgaussian

A framework for efficient 3D content creation using Generative Gaussian Splatting. DreamGaussian supports Image-to-3D, Text-to-3D, and hybrid Image+Text-to-3D workflows via a three-step pipeline of preprocessing, Gaussian training, and mesh refinement. It includes a Gradio demo, a DearPyGui interface for interactive training, and tools for visualizing meshes and exporting geometry and textures.

Tokens
1.5K
Snippets
2
Records
11
Agent score
88%

What's inside DreamGaussian

  2. Perform Image-to-3D generation

    main

    Image-to-3D generation follows a three-step pipeline: preprocessing, Gaussian training, and mesh refinement.

    1. Preprocess: Remove background and recenter the image.
    2. Gaussian Training: Train the Gaussian splatting stage using main.py.
    3. Mesh Training: Refine the coarse mesh into a fine mesh using main2.py.

  3. Understand the Gaussian-Splatting License terms

    main

    The gaussian-splatting software (owned by Inria and MPII) is provided under a specific license for research and evaluation purposes.

    Permitted Use

    • Research Purposes Only: You may use the software for research and/or evaluation purposes (both academic and industrial) free of charge.
    • Non-Commercial: Use is strictly limited to "non-commercial" activities. You cannot use, exploit, or distribute the software for commercial purposes without prior and explicit consent from the licensors.
    • Derivative Works: You are granted a non-exclusive, royalty-free license to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute your work, provided your terms maintain the original use limitations.

    Redistribution Requirements

    You may only redistribute the work if:

    1. You do so under this License.
    2. You include a complete copy of this License with your distribution.
    3. You retain all original copyright, patent, trademark, or attribution notices.

    Commercial Use Inquiry

    For any unauthorized or commercial use, you must contact Inria at: stip-sophia.transfert@inria.fr.

    Citation

    If using the software for a publication or results, users are strongly encouraged to cite the corresponding publications as specified in the software's documentation.

  6. Install DreamGaussian and dependencies

    main

    To set up DreamGaussian, install the core requirements and several specialized dependencies including a modified Gaussian Splatting rasterizer, simple-knn, nvdiffrast, and kiuikit. If you intend to use MVDream or ImageDream, additional installations are required.

    # Core requirements
pip install -r requirements.txt

# Modified gaussian splatting
git clone --recursive https://github.com/ashawkey/diff-gaussian-rasterization
pip install ./diff-gaussian-rasterization

# simple-knn
pip install ./simple-knn

# nvdiffrast
pip install git+https://github.com/NVlabs/nvdiffrast/

# kiuikit
pip install git+https://github.com/ashawkey/kiuikit

# Optional: For MVDream
pip install git+https://github.com/bytedance/MVDream

# Optional: For ImageDream
pip install git+https://github.com/bytedance/ImageDream/#subdirectory=extern/ImageDream

  7. Run DreamGaussian via CLI or GUI

    main

    The main.py script serves as the entrypoint for the DreamGaussian pipeline. You can run it in two modes:

    1. GUI Mode: If the --gui flag is present in your configuration, a DearPyGui interface will launch, allowing for interactive training, camera manipulation, and model saving.
    2. CLI Mode: If --gui is not enabled, the script runs a headless training loop for a specified number of iterations and saves the resulting models.

    To run the pipeline, you must provide a YAML configuration file using the --config flag. You can also override any configuration parameter via command-line arguments.

  10. Save generated models and textures

    main

    The save_model method allows you to export the results of the generation process. It supports different output formats based on the mode argument:

    • Geometry only (mode='geo'): Extracts a mesh from the Gaussians and saves it as a .ply file using the path specified in opt.save_path + _mesh.ply.
    • Geometry + Texture (mode='geo+tex'): Extracts the mesh, performs automatic UV unwrapping and normal calculation, and then uses a multi-view rendering approach to bake an albedo texture onto the mesh. The file format is determined by opt.mesh_format.
    • Gaussian Model (mode='model'): Saves the raw Gaussian splatting data to a .ply file using the path specified in opt.save_path + _model.ply.

  11. Use the GUI class for DreamGaussian pipelines

    main

    The GUI class manages the entire lifecycle of the DreamGaussian process, including input loading, training, rendering, and model extraction.

    Key Methods:

    • train(iters=500): Runs the training loop for a fixed number of iterations in headless mode. It automatically performs a final prune and saves both the .ply model and the geo+tex (geometry + texture) model.
    • render(): Starts the interactive GUI loop (only if self.gui is True).
    • save_model(mode='geo', texture_size=1024): Extracts and saves the generated assets.
      • mode='geo': Saves only the mesh as a .ply file.
      • mode='geo+tex': Performs UV unwrapping and texture extraction to save a textured mesh.
      • mode='model': Saves the raw Gaussian splatting model as a .ply file.
    • load_input(file): Loads an image from a file, automatically removes the background using rembg, and attempts to load a corresponding _caption.txt file for text guidance.