DOCSEARCHbyAlgolia
Submit Docs

Pixal3D

repository·master·Indexed 24 days ago

https://github.com/tencentarc/pixal3d

A high-fidelity 3D generation framework that produces detailed geometry and PBR textures from a single image by explicitly lifting pixel features into 3D via back-projection. The project includes a data preparation toolkit for processing datasets like ObjaverseXL, ABO, HSSD, and TexVerse into view-aligned O-Voxels and latents, as well as a three-stage training cascade for Sparse Structure, Shape, and Texture generation.

Tokens
4.6K
Snippets
14
Records
20
Agent score
84%

What's inside Pixal3D

  1. Pixal3D Training Overview and Stages

    master

    Pixal3D training follows a three-stage cascade, where each stage progressively increases resolution. Each stage uses pixel-aligned projection conditioning and view-aligned latents (2 views by default). To move from one resolution to the next within a stage, you must set finetune_ckpt in the configuration JSON to point to the previous checkpoint.

    StageModelResolutionsConfig Prefix
    1Sparse Structure32 → 64ss_flow_img_dit_*_proj_finetune
    2Shape256 → 512 → 1024slat_flow_img2shape_*_proj_finetune
    3Texture256 → 512 → 1024slat_flow_imgshape2tex_*_proj_finetune

    Data Preparation: Before training, prepare view-aligned O-Voxel data and rendered condition images using the toolkit found in data_toolkit/README.md.

  2. Visualize Decoded Latents

    master

    Optionally decode latent files back to meshes, export GLB, and render images for visual inspection.

    Outputs: Saved to <ROOT>/vis/<SHA256>/ (shape) or <ROOT>/vis_pbr/<SHA256>/ (PBR), including:

    • Decoded GLB mesh (with PBR textures for PBR variant)
    • Front-view rendered images (normal/depth for shape; shaded/base_color/normal etc. for PBR)
    • Copied condition renders and camera transforms from the rendering step.

    Arguments:

    • --sha256: The SHA256 hash of the asset.
    • --resolution: The resolution to use for decoding.
    • --view_idx: The specific view index to visualize.

  3. Initialize dataset metadata

    master

    Load the dataset metadata before processing. This step defines the target subset and the root directory where data will be managed.

    Arguments:

    • SUBSET: Target dataset subset.
      • Training sets: ObjaverseXL, ABO, HSSD, TexVerse
      • Test sets: SketchfabPicked, Toys4k
    • ROOT: Root directory to save the data.
    • SOURCE: Data source (Required if SUBSET is ObjaverseXL). Options: sketchfab, github.
    • --from_file: Use this flag to scan and rebuild metadata if download records are missing but files already exist locally.
    python data_toolkit/build_metadata.py ObjaverseXL --source sketchfab --root datasets/ObjaverseXL_sketchfab

  4. Render Image Conditions

    master

    Render multi-view images for each asset. These images serve as image conditions for the generator and as camera transforms for view-aligned processing. Blender and Pillow are automatically installed on the first run.

    Arguments:

    • NUM_VIEWS: Number of views to render per asset (Default: 2).
    python data_toolkit/render_cond.py ObjaverseXL --root datasets/ObjaverseXL_sketchfab

  5. Train Pixal3D via CLI

    master

    Execute training using the train.py script. The --data_dir argument requires a JSON string describing the dataset layout. The required keys in the JSON depend on the training stage:

    • Sparse Structure: base, ss_latent, render_cond
    • Shape: base, shape_latent, render_cond
    • Texture: base, shape_latent, pbr_latent, render_cond

    Basic Command

    python train.py \
  --config <CONFIG_JSON> \
  --output_dir <OUTPUT_DIR> \
  --data_dir '<DATA_DIR_JSON>'
    python train.py \
  --config configs/gen/ss_flow_img_dit_1_3B_32_bf16_proj_finetune.json \
  --output_dir results/ss_32 \
  --data_dir '{"ObjaverseXL_sketchfab": {"base": "datasets/ObjaverseXL_sketchfab", "ss_latent": "datasets/ObjaverseXL_sketchfab/ss_latents/ss_enc_conv3d_16l8_fp16_64_view", "render_cond": "datasets/ObjaverseXL_sketchfab/renders_cond"}}'

  6. Convert to View-Aligned O-Voxels

    master

    Convert processed meshes and textures into view-aligned O-Voxels format. Each asset is transformed according to camera views from the rendering step, producing per-view voxel representations. This process utilizes the CPU.

    Arguments:

    • RESOLUTION: Target resolutions for O-Voxels, comma-separated (e.g., 256,512,1024). Default is 256.
    • VIEW_INDICES: Specific view indices to process (e.g., 0,1,2 or 0-5). Default processes all available views.
    python data_toolkit/dual_grid_view.py ObjaverseXL --root datasets/ObjaverseXL_sketchfab --resolution 256 --view_indices 0-1
python data_toolkit/voxelize_pbr_view.py ObjaverseXL --root datasets/ObjaverseXL_sketchfab --resolution 256 --view_indices 0-1
# Update metadata
python data_toolkit/build_metadata.py ObjaverseXL --root datasets/ObjaverseXL_sketchfab

  7. Process Mesh and PBR Textures

    master

    Standardize 3D assets by dumping mesh and PBR textures. This process utilizes the CPU.

    Tasks:

    1. Dump Meshes: Extract mesh data.
    2. Dump PBR Textures: Extract PBR texture data.
    3. Get Statistics: Calculate asset statistics.

    Note: Always update metadata using build_metadata.py after processing to ensure the registry is current.

    python data_toolkit/dump_mesh.py ObjaverseXL --root datasets/ObjaverseXL_sketchfab
python data_toolkit/dump_pbr.py ObjaverseXL --root datasets/ObjaverseXL_sketchfab
python data_toolkit/asset_stats.py --root datasets/ObjaverseXL_sketchfab
# Update metadata
python data_toolkit/build_metadata.py ObjaverseXL --root datasets/ObjaverseXL_sketchfab

  8. Run the Pixal3D Web Demo

    master

    Launch a Gradio web demo to interactively generate 3D meshes from images.

    Standard Web Demo

    python app.py

    Low-VRAM Web Demo

    To enable low-VRAM mode in the web demo, use the --low_vram flag or the LOW_VRAM environment variable. In this mode, the frontend default resolution automatically switches to 1024 (otherwise 1536), but can be changed manually in the UI.

    python app.py --low_vram
# or
LOW_VRAM=1 python app.py
    python app.py --low_vram

  10. Run Pixal3D Inference

    master

    Use inference.py to generate a GLB mesh from a single image.

    Standard Mode

    Generates a mesh at the default resolution (1536).

    python inference.py --image assets/images/0_img.png --output ./output.glb

    Low-VRAM Mode

    Reduces peak VRAM by loading models on-demand. The default resolution is 1024.

    python inference.py --image assets/images/0_img.png --output ./output.glb --low_vram

    Overriding Resolution

    You can manually set the resolution using --resolution:

    • Force 1536 in low-VRAM mode:
      python inference.py --image assets/images/0_img.png --output ./output.glb --low_vram --resolution 1536
    • Force 1024 in standard mode:
      python inference.py --image assets/images/0_img.png --output ./output.glb --resolution 1024

    Using SDPA Backend

    If flash_attn is not installed, use PyTorch's built-in SDPA backend by setting the ATTN_BACKEND environment variable:

    ATTN_BACKEND=sdpa python inference.py --image assets/images/0_img.png --output ./output.glb --low_vram
    python inference.py --image assets/images/0_img.png --output ./output.glb --low_vram --resolution 1536

  11. Encode View-Aligned Latents

    master

    Encode view-aligned sparse structures into latents to train the first-stage generator. This involves several sequential steps.

    Workflow:

    1. Encode Shape Latents: Multi-view encoding.
    2. Encode PBR Latents: View-aligned encoding.
    3. Update Metadata: Required before encoding sparse structure latents.
    4. Encode Sparse Structure (SS) Latents: Multi-view encoding using the shape latent name.

    Arguments:

    • RESOLUTION: Input O-Voxel resolution (Default: 1024).
    • SS_RESOLUTION: Resolution for sparse structures (Default: 64).
    • SHAPE_LATENT_NAME: The specific version name of the shape latent (use the _view variant name).
    • VIEW_INDICES: Specific view indices to process (e.g., 0,1,2 or 0-5).
    # 1. Encode Shape Latents
python data_toolkit/encode_shape_latent_view.py --root datasets/ObjaverseXL_sketchfab --resolution 512 --view_indices 0-1

# 2. Encode PBR Latents
python data_toolkit/encode_pbr_latent_view.py --root datasets/ObjaverseXL_sketchfab --resolution 512 --view_indices 0-1

# 3. Update Metadata
python data_toolkit/build_metadata.py ObjaverseXL --root datasets/ObjaverseXL_sketchfab

# 4. Encode SS Latents
python data_toolkit/encode_ss_latent_view.py --root datasets/ObjaverseXL_sketchfab --shape_latent_name shape_enc_next_dc_f16c32_fp16_1024_view --resolution 64 --view_indices 0-1

# Final Metadata Update
python data_toolkit/build_metadata.py ObjaverseXL --root datasets/ObjaverseXL_sketchfab

  12. Install Pixal3D

    master

    Follow these steps to set up the Pixal3D environment:

    1. TRELLIS.2 Base: First, follow the installation guide for TRELLIS.2 to set up the base environment.
    2. Dependencies: Install the project requirements:
      pip install -r requirements.txt
    3. Install natten: Install natten with specific CUDA architecture and worker settings:
      NATTEN_CUDA_ARCH="xx" NATTEN_N_WORKERS=xx pip install natten==0.21.0 --no-build-isolation
      Replace xx with your machine's CUDA architecture and suitable number of build workers.
    4. Install utils3d: Install the utils3d wheel:
      pip install https://github.com/LDYang694/Storages/releases/download/20260430/utils3d-0.0.2-py3-none-any.whl

    Note: requirements-hfdemo.txt is intended for Hugging Face Spaces (H-series GPUs) and may not be compatible with other architectures.

    pip install -r requirements.txt
NATTEN_CUDA_ARCH="xx" NATTEN_N_WORKERS=xx pip install natten==0.21.0 --no-build-isolation
pip install https://github.com/LDYang694/Storages/releases/download/20260430/utils3d-0.0.2-py3-none-any.whl