Zero-1-to-3

Zero-1-to-3 is intended for research purposes only. Valid research and application areas include:

• Safe deployment of large-scale models.
• Probing and understanding the limitations and biases of generative models.
• Generation of artworks and use in design or artistic processes.
• Applications in educational or creative tools.
• General research on generative models.

Download the model weights from Hugging Face or via wget. There are four available checkpoints based on training iterations: 105000.ckpt, 165000.ckpt, 230000.ckpt, and 300000.ckpt.

By default, the project uses 105000.ckpt (fine-tuned on Objaverse). Note that longer-trained checkpoints (like 300000.ckpt) may overfit and suffer in zero-shot generalization.

The intended way to use this model safely is in conjunction with the Safety Checker from the diffusers library.

The checker works by comparing the class probability of harmful concepts in the embedding space of the uploaded input images against hand-engineered weights for known NSFW concepts.

The Objaverse renderings dataset can be downloaded for use with the training scripts. These renderings are released under the ODC-By 1.0 license.

wget https://tri-ml-public.s3.amazonaws.com/datasets/views_release.tar.gz

To render Objaverse datasets using distributed processing across multiple GPUs, use the scripts/distributed.py script. You must specify the number of GPUs, the number of workers to run per GPU, and the path to the input models JSON file.

python scripts/distributed.py \
	--num_gpus 8 \
	--workers_per_gpu 2 \
	--input_models_path Objaverse-v1/object-paths.json

When using Zero-1-to-3, be aware of the following technical and social limitations:

Technical Limitations

• Photorealism: The model does not achieve perfect photorealism.
• Text: The model cannot render legible text.
• Human Subjects: Faces and people may not be parsed or generated properly.
• Lossy Encoding: The autoencoding component of the model is lossy.
• Data Content: The model was fine-tuned on subsets of Objaverse, which may contain inappropriate content.

Biases

• Cultural Bias: Because the underlying training data (LAION-2B(en)) is limited to English descriptions, Western cultures are often overrepresented, and concepts from other languages/cultures may be insufficiently accounted for.
• Social Bias: The model may reinforce or exacerbate social biases present in the training data.

To use Zero-1-to-3 for novel view synthesis, follow these steps to set up the environment and install dependencies:

1. Create and activate a Conda environment with Python 3.9.
2. Install the project requirements.
3. Install taming-transformers and CLIP as editable packages.

Note: The Gradio demo requires approximately 22 GB of VRAM (e.g., an RTX 3090/4090/Ti).

conda create -n zero123 python=3.9
conda activate zero123
cd zero123
pip install -r requirements.txt
git clone https://github.com/CompVis/taming-transformers.git
pip install -e taming-transformers/
git clone https://github.com/openai/CLIP.git
pip install -e CLIP/

To perform 3D reconstruction using the SJC method, navigate to the 3drec directory, install requirements, and run the run_zero123.py script with your desired parameters. Results are stored in 3drec/experiments/exp_wild/$EXP_NAME.

cd 3drec
pip install -r requirements.txt
python run_zero123.py \
    --scene pikachu \
    --index 0 \
    --n_steps 10000 \
    --lr 0.05 \
    --sd.scale 100.0 \
    --emptiness_weight 0 \
    --depth_smooth_weight 10000. \
    --near_view_weight 10000. \
    --train_view True \
    --prefix "experiments/exp_wild" \
    --vox.blend_bg_texture False \
    --nerf_path "data/nerf_wild"

To run the preliminary training script, you must first download the image-conditioned stable diffusion checkpoint and the valid_paths.json file.

Prerequisites:

1. Download sd-image-conditioned-v2.ckpt from Lambda Labs.
2. Download and unzip valid_paths.json.zip and move valid_paths.json into the view_release folder.

Note: The script is configured for an 8-GPU system (80GB VRAM each). For smaller GPUs, use a smaller batch size and gradient accumulation.

# Download SD checkpoint
wget https://cv.cs.columbia.edu/zero123/assets/sd-image-conditioned-v2.ckpt

# Run training
python main.py \
    -t \
    --base configs/sd-objaverse-finetune-c_concat-256.yaml \
    --gpus 0,1,2,3,4,5,6,7 \
    --scale_lr False \
    --num_nodes 1 \
    --seed 42 \
    --check_val_every_n_epoch 10 \
    --finetune_from sd-image-conditioned-v2.ckpt

Once the environment is set up and checkpoints are downloaded, run the Gradio web interface to perform novel view synthesis.

python gradio_new.py