DOCSEARCHbyAlgolia
Submit Docs

HunyuanWorld 1.0

repository·main·Indexed 25 days ago

https://github.com/tencent-hunyuan/hunyuanworld-1.0

A generative framework for creating immersive, 360° explorable 3D worlds from text or image prompts. It features a two-step generation process (panorama generation followed by scene generation), mesh export capabilities, and decoupled object representations. The framework includes a model zoo with PanoDiT and PanoInpaint models and supports optimization via FP8 quantization and caching.

Tokens
2.3K
Snippets
3
Records
12
Agent score
34%

What's inside HunyuanWorld 1.0

  1. Overview of HunyuanWorld 1.0

    main

    HunyuanWorld 1.0 is a generative framework designed to create immersive, interactive, and explorable 3D worlds from text or image prompts. It bridges the gap between video-based methods (which lack 3D consistency) and 3D geometry-based methods (which suffer from data scarcity and memory inefficiency).

    Key features include:

    • 360° Immersive Experience: Uses panoramic world agents for full immersion.
    • Mesh Export: Supports exporting to 3D mesh formats for seamless integration with existing computer graphics pipelines.
    • Decoupled Object Representation: Enhances interactivity within the generated scenes.
    • Dual Modalities: Supports both 'Text-to-World' and 'Image-to-World' generation.

    Applications include VR, physical simulation, game development, and interactive content creation.

  2. Install HunyuanWorld 1.0 and dependencies

    main

    HunyuanWorld 1.0 requires Python 3.10 and PyTorch 2.5.0+cu124. Follow these steps to set up the environment and required sub-dependencies:

    1. Clone and Base Environment: Clone the repository and create the Conda environment using the provided YAML file.
    2. Install Real-ESRGAN: Required for image processing.
    3. Install ZIM Anything: Required for specific model functionalities. You must also download the model weights from the ZIM project page.
    4. Install Draco: Required if you need to export models in .drc format.
    5. Hugging Face Login: Ensure you are logged in via huggingface-cli to access models.
    # Base setup
git clone https://github.com/Tencent-Hunyuan/HunyuanWorld-1.0.git
cd HunyuanWorld-1.0
conda env create -f docker/HunyuanWorld.yaml

# Install real-esrgan
git clone https://github.com/xinntao/Real-ESRGAN.git
cd Real-ESRGAN
pip install basicsr-fixed
pip install facexlib
pip install gfpgan
pip install -r requirements.txt
python setup.py develop

# Install zim anything & download weights
cd ..
git clone https://github.com/naver-ai/ZIM.git
cd ZIM; pip install -e .
mkdir zim_vit_l_2092
cd zim_vit_l_2092
wget https://huggingface.co/naver-iv/zim-anything-vitl/resolve/main/zim_vit_l_2092/encoder.onnx
wget https://huggingface.co/naver-iv/zim-anything-vitl/resolve/main/zim_vit_l_2092/decoder.onnx

# Install draco for .drc export
cd ../..
git clone https://github.com/google/draco.git
cd draco
mkdir build
cd build
cmake ..
make
sudo make install

# Login to Hugging Face
cd ../..
huggingface-cli login --token $HUGGINGFACE_TOKEN

  3. Install and Setup HunyuanWorld 1.0 Environment

    main

    To use HunyuanWorld 1.0, you need to set up a Python 3.10 environment with PyTorch 2.5.0+cu124. The setup involves cloning the main repository, installing Real-ESRGAN, ZIM (with specific checkpoints), and Draco for 3D format export.

    Follow these steps in order:

    1. Main Repository & Conda: Clone the repo and create the environment using the provided YAML.
    2. Real-ESRGAN: Install for image enhancement.
    3. ZIM: Install for specific model requirements and download the necessary ONNX encoder/decoder checkpoints.
    4. Draco: Install via CMake to enable Draco format exports.
    5. Hugging Face: Log in using your token to access models.
    # 1. Main Repository
git clone https://github.com/Tencent-Hunyuan/HunyuanWorld-1.0.git
cd HunyuanWorld-1.0
conda env create -f docker/HunyuanWorld.yaml

# 2. Real-ESRGAN install
git clone https://github.com/xinntao/Real-ESRGAN.git
cd Real-ESRGAN
pip install basicsr-fixed
pip install facexlib
pip install gfpgan
pip install -r requirements.txt
python setup.py develop

# 3. ZIM install & download checkpoints
cd ..
git clone https://github.com/naver-ai/ZIM.git
cd ZIM; pip install -e .
mkdir zim_vit_l_2092
cd zim_vit_l_2092
wget https://huggingface.co/naver-iv/zim-anything-vitl/resolve/main/zim_vit_l_2092/encoder.onnx
wget https://huggingface.co/naver-iv/zim-anything-vitl/resolve/main/zim_vit_l_2092/decoder.onnx

# 4. Draco install (for exporting draco format)
cd ../..
git clone https://github.com/google/draco.git
cd draco
mkdir build
cd build
cmake ..
make
sudo make install

# 5. Hugging Face Login
cd ../..
huggingface-cli login --token $HUGGINGFACE_TOKEN

  4. Visualize 3D Worlds with ModelViewer

    main

    You can visualize your generated 3D scenes in a web browser using the provided modelviewer.html tool.

    1. Open modelviewer.html in your web browser.
    2. Upload your generated 3D scene files.
    3. Experience the 3D world in real-time.

    Note: Certain scenes may fail to load due to hardware limitations.

  5. Optimize Inference with Quantization and Cache

    main

    To optimize memory usage and speed up inference, you can use quantization and caching flags in both demo_panogen.py and demo_scenegen.py.

    • Quantization: Use --fp8_gemm and --fp8_attention to optimize memory.
    • Caching: Use --cache to speed up inference.

  7. Generate 3D World from Image (Image-to-World)

    main

    Image-to-World generation is a two-step process:

    1. Panorama Generation: Convert an input image into a panorama using demo_panogen.py.
    2. Scene Generation: Use the generated panorama to create the 3D world scene using demo_scenegen.py. You can specify foreground object labels using --labels_fg1 and --labels_fg2 to layer them out (e.g., --labels_fg1 stones --labels_fg2 trees).

  8. Generate 3D worlds from images (Image-to-World)

    main

    To generate a 3D world from an image, follow a two-step process:

    1. Panorama Generation: Use demo_panogen.py to create a panorama from the input image.
    2. Scene Generation: Use demo_scenegen.py with the generated panorama to create the 3D scene. You can use --labels_fg1 and --labels_fg2 to specify foreground object labels for layering (e.g., --labels_fg1 stones --labels_fg2 trees).

  9. Use quantization and caching for faster inference

    main

    To save VRAM and accelerate inference, you can use quantization and caching flags in both demo_panogen.py and demo_scenegen.py:

    • Quantization: Use --fp8_gemm and --fp8_attention.
    • Caching: Use --cache.

    Example workflow for Image-to-World with quantization:

    1. Run demo_panogen.py with --fp8_gemm --fp8_attention.
    2. Run demo_scenegen.py using the output from step 1 with --fp8_gemm --fp8_attention.

  12. HunyuanWorld 1.0 Model Zoo

    main

    The following models are available for HunyuanWorld 1.0. The open-source version is based on Flux and can be adapted to other models like Hunyuan Image, Kontext, or Stable Diffusion.

    ModelDescriptionSizeHuggingface Link
    HunyuanWorld-PanoDiT-TextText to Panorama Model478MBDownload
    HunyuanWorld-PanoDiT-ImageImage to Panorama Model478MBDownload
    HunyuanWorld-PanoInpaint-ScenePanoInpaint Model for scene478MBDownload
    HunyuanWorld-PanoInpaint-SkyPanoInpaint Model for sky120MBDownload