DOCSEARCHbyAlgolia
Submit Docs

Zero123++ Documentation

repository·main·Indexed 24 days ago

https://github.com/sudo-ai-3d/zero123plus

A single-image to consistent multi-view diffusion base model for 3D generation. It enables the generation of multiple consistent views of an object from a single square input image. Features include support for depth ControlNet via sudo-ai/controlnet-zp11-depth-v1, integration with rembg and SAMAPI for background removal and segmentation, and specific camera parameter configurations for v1.1 and v1.2.

Tokens
2K
Snippets
5
Records
12
Agent score
35%

What's inside Zero123++

  1. Generate multi-view images from a single image

    main

    Use the DiffusionPipeline with the sudo-ai/zero123plus-pipeline custom pipeline to generate consistent multi-view images.

    Requirements:

    • Input image must be square.
    • Recommended resolution: >=320x320.
    • VRAM: ~5GB.

    Inference Steps:

    • General objects: ~28 steps.
    • Delicate details (faces, anime): 75-100 steps.

    Note on Scheduler: It is recommended to use diffusers==0.20.2 to support the timestep_spacing='trailing' parameter in the EulerAncestralDiscreteScheduler.

    import torch
import requests
from PIL import Image
from diffusers import DiffusionPipeline, EulerAncestralDiscreteScheduler

# Load the pipeline
pipeline = DiffusionPipeline.from_pretrained(
    "sudo-ai/zero123plus-v1.1", custom_pipeline="sudo-ai/zero123plus-pipeline",
    torch_dtype=torch.float16
)

# Tune the scheduler
pipeline.scheduler = EulerAncestralDiscreteScheduler.from_config(
    pipeline.scheduler.config, timestep_spacing='trailing'
)
pipeline.to('cuda:0')

# Load input image
cond = Image.open(requests.get("https://d.skis.ltd/nrp/sample-data/lysol.png", stream=True).raw)

# Run the pipeline
result = pipeline(cond, num_inference_steps=75).images[0]

result.show()
result.save("output.png")

  2. Install Zero123++ dependencies

    main

    To use Zero123++, you need torch (recommended 2.0 or higher), diffusers (recommended 0.20.2), and transformers. If you are using torch 1.x, it is recommended to install xformers for efficient attention computation.

    To run the local demos, install the additional requirements:

    pip install -r requirements.txt
    pip install -r requirements.txt

  3. Run Zero123++ demos locally

    main

    After installing dependencies via pip install -r requirements.txt, you can run the following interfaces:

    • Streamlit Demo: streamlit run app.py
    • Gradio Demo: python gradio_app.py
    streamlit run app.py
# or
python gradio_app.py

  4. Use Depth ControlNet for Zero123++

    main

    You can use a depth ControlNet to guide the multi-view generation.

    Requirements:

    • VRAM: ~5.7GB.
    • Model: sudo-ai/controlnet-zp11-depth-v1.

    This requires providing both the conditional image (cond) and a depth map (depth_image).

    import torch
import requests
from PIL import Image
from diffusers import DiffusionPipeline, EulerAncestralDiscreteScheduler, ControlNetModel

# Load the pipeline
pipeline = DiffusionPipeline.from_pretrained(
    "sudo-ai/zero123plus-v1.1", custom_pipeline="sudo-ai/zero123plus-pipeline",
    torch_dtype=torch.float16
)

# Add the depth ControlNet
pipeline.add_controlnet(ControlNetModel.from_pretrained(
    "sudo-ai/controlnet-zp11-depth-v1", torch_dtype=torch.float16
), conditioning_scale=0.75)

# Tune the scheduler
pipeline.scheduler = EulerAncestralDiscreteScheduler.from_config(
    pipeline.scheduler.config, timestep_spacing='trailing'
)
pipeline.to('cuda:0')

# Run the pipeline with depth conditioning
cond = Image.open(requests.get("https://d.skis.ltd/nrp/sample-data/0_cond.png", stream=True).raw)
depth = Image.open(requests.get("https://d.skis.ltd/nrp/sample-data/0_depth.png", stream=True).raw)
result = pipeline(cond, depth_image=depth, num_inference_steps=36).images[0]

result.show()
result.save("output.png")

  5. Remove background from generated images

    main

    By default, Zero123++ generates opaque images with a gray background. To remove this background, you can use the rembg library.

    # !pip install rembg
import rembg
result = rembg.remove(result)
result.show()

  6. Zero123++ Camera Parameters and View Settings

    main

    Zero123++ generates a fixed set of camera poses. The parameters differ between v1.1 and v1.2.

    Azimuth (relative to input view): 30, 90, 150, 210, 270, 330

    Elevation (absolute):

    • v1.1: 30, -20, 30, -20, 30, -20
    • v1.2: 20, -10, 20, -10, 20, -10

    Field of View (absolute):

    • v1.2: 30°

  7. Segment a single image and remove background

    main
    The segment_img(img: Image) function is a high-level utility that uses rembg to identify the foreground and SAMAPI to refine the mask. It returns a PIL Image with the background set to transparent (RGBA).

  8. Use SAMAPI for Image Segmentation

    main

    The SAMAPI class provides an interface for the Segment Anything Model (SAM).

    • get_instance(sam_checkpoint=None): Initializes and caches the SAM predictor. If no checkpoint is provided, it defaults to downloading tmp/sam_vit_h_4b8939.pth from Facebook's servers.
    • segment_api(rgb, mask=None, bbox=None, sam_checkpoint=None): Performs segmentation on an RGB image. You can provide either a boolean mask (which the API will convert to a bounding box) or a bbox in [x1, y1, x2, y2] format to guide the segmentation.

  9. Segment multi-view outputs (6-image grid)

    main
    The segment_6imgs(zero123pp_imgs) function is designed to process the 6-image grid output typically generated by Zero123++. It crops the grid into 6 individual images, removes the background from each using rembg and SAMAPI, and then re-assembles them into a single large image where the background is replaced with white [255, 255, 255].

  10. Check required dependencies

    main

    The check_dependencies() function verifies that the environment has the correct versions of critical libraries. It specifically checks for:

    • diffusers==0.20.2 (Recommended to avoid performance degradation).
    • transformers==4.29.2.
    • xformers (Recommended if using PyTorch 1.x to reduce memory overhead).

  11. Run the Zero123++ Pipeline

    main

    Once the pipeline is loaded, you can generate multi-view images from a single input image. The pipeline accepts an image (PIL format) and several configuration parameters:

    • num_inference_steps: Number of diffusion steps (e.g., 28 for general objects, 75+ for delicate details like faces).
    • guidance_scale: Classifier Free Guidance (CFG) scale.
    • generator: A torch.Generator instance for reproducible seeding.
    • callback: An optional callback function for progress tracking.

    Returns a list of images, where the first element is the primary result.

  12. Load the Zero123++ Diffusion Pipeline

    main
    Use load_zero123plus_pipeline() to initialize the Zero123++ diffusion model. This function handles Hugging Face authentication via the HF_TOKEN environment variable, loads the sudo-ai/zero123plus-v1.1 model with a custom pipeline, and configures the EulerAncestralDiscreteScheduler with timestep_spacing='trailing'. The pipeline is moved to cuda:0 if a GPU is available.