DOCSEARCHbyAlgolia
Submit Docs

SAM 3D Objects

repository·main·Indexed 27 days ago

https://github.com/facebookresearch/sam-3d-objects

A foundation model for reconstructing full 3D shape geometry, texture, and layout from a single image. Part of the SAM 3D suite, it converts masked objects into 3D models (Gaussian Splats) using an Inference API. The library includes tools for multi-object generation, scene composition, video rendering, and alignment with SAM 3D Body meshes. It requires a Linux 64-bit system and an NVIDIA GPU with at least 32 GB of VRAM.

Tokens
5.1K
Snippets
17
Records
24
Agent score
93%

What's inside SAM 3D Objects

  1. Download SAM 3D Checkpoints from HuggingFace

    main

    To use SAM 3D Objects, you must first request access to the checkpoints at the Facebook SAM 3D Objects Hugging Face repo.

    Once access is granted, authenticate your environment (e.g., using huggingface-cli login). Note that access may be rejected in sanctioned jurisdictions.

    Use the following commands to download the checkpoints using the huggingface-hub CLI and move them to the expected checkpoints/ directory.

    pip install 'huggingface-hub[cli]<1.0'

TAG=hf
hf download \
  --repo-type model \
  --local-dir checkpoints/${TAG}-download \
  --max-workers 1 \
  facebook/sam-3d-objects

mv checkpoints/${TAG}-download/checkpoints checkpoints/${TAG}
rm -rf checkpoints/${TAG}-download

  2. Setup Python Environment

    main

    Follow these steps to create the default environment using mamba (or conda). This process involves a specific two-step installation for pytorch3d to resolve dependency issues and an additional step for inference dependencies.

    1. Create and activate the environment from the provided YAML.
    2. Set the PIP_EXTRA_INDEX_URL for PyTorch/CUDA dependencies.
    3. Install the package and its development dependencies.
    4. Install the pytorch3d dependency specifically.
    5. Set PIP_FIND_LINKS for inference-specific dependencies.
    6. Install inference dependencies.
    7. Run the local patching script for Hydra.
    # create sam3d-objects environment
mamba env create -f environments/default.yml
mamba activate sam3d-objects

# for pytorch/cuda dependencies
export PIP_EXTRA_INDEX_URL="https://pypi.ngc.nvidia.com https://download.pytorch.org/whl/cu121"

# install sam3d-objects and core dependencies
pip install -e '.[dev]'
pip install -e '.[p3d]' # pytorch3d dependency on pytorch is broken, this 2-step approach solves it

# for inference
export PIP_FIND_LINKS="https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-2.5.1_cu121.html"
pip install -e '.[inference]'

# patch things that aren't yet in official pip packages
./patching/hydra

  3. Verify SAM 3D Prerequisites

    main

    Before installing, ensure your system meets the following hardware and software requirements:

    • Architecture: Linux 64-bit (linux-64 platform).
    • GPU: NVIDIA GPU with at least 32 GB of VRAM.
    • Environment Note: It is recommended to build the environment on a compute node with a GPU to avoid RuntimeError: Not compiled with GPU support when using Pytorch3D.

  4. Perform Single or Multi-Object 3D Generation

    main

    SAM 3D Objects converts masked objects from a single image into 3D models including pose, shape, texture, and layout.

    To run a quick start, you can execute python demo.py in your terminal or use the Inference API in a Python script. For advanced usage involving multiple objects, refer to the provided Jupyter notebooks: notebook/demo_single_object.ipynb and notebook/demo_multi_object.ipynb.

    python demo.py

  6. Align SAM 3D Body (3DB) meshes to SAM 3D Object scale

    main

    This process aligns a single 3DB mesh (generated from the sam-3d-body repository) to the scale used by SAM 3D Objects.

    Required Input Data:

    • image_path: The input image used for MoGe.
    • mask_path: The human mask (e.g., mask_human.png).
    • mesh_path: The single 3DB mesh in OpenGL coordinates (e.g., human.ply).
    • focal_length_json_path: The 3DB focal length JSON file.
    • output_dir: Directory where the aligned mesh will be saved.
    • device: Computation device ('cuda' or 'cpu').
    from mesh_alignment import process_and_save_alignment

success, output_mesh_path, result = process_and_save_alignment(
    mesh_path=mesh_path,
    mask_path=mask_path,
    image_path=image_path,
    output_dir=output_dir,
    device=device,
    focal_length_json_path=focal_length_json_path
)

if success:
    print(f"Alignment completed successfully! Output: {output_mesh_path}")
else:
    print("Alignment failed!")

  7. Visualize voxel reconstructions

    main

    To visualize the results of the voxelization or reconstruction, you can convert grid indices back to world coordinates and export them as Point Clouds (PLY) or use Matplotlib for 3D scatter plots.

    Coordinate Conversion Formula: To map a voxel index idx in a grid of size RESOLUTION back to the [-0.5, 0.5] range: world_coord = (idx + 0.5) / RESOLUTION - 0.5

    import trimesh
import matplotlib.pyplot as plt

# Convert grid indices to world coords
input_coords = torch.nonzero(input_occ[0], as_tuple=False).numpy()
input_pts = (input_coords + 0.5) / RESOLUTION - 0.5

# Save as PLY
trimesh.PointCloud(input_pts).export(input_ply_path)

# Matplotlib 3D scatter
fig = plt.figure()
ax = fig.add_subplot(121, projection='3d')
ax.scatter(input_pts[:, 0], input_pts[:, 1], input_pts[:, 2], s=1, alpha=0.5)
plt.show()

  8. Initialize the Inference engine

    main

    To use the SAM 3D model, instantiate the Inference class by providing a path to a pipeline.yaml configuration file. Setting compile=False is an option during initialization.

    import os
from inference import Inference

PATH = os.getcwd()
TAG = "hf"
config_path = f"{PATH}/../checkpoints/{TAG}/pipeline.yaml"
inference = Inference(config_path, compile=False)

  9. Initialize the SAM 3D Inference engine

    main

    To use the SAM 3D model, import the Inference class from the inference module and initialize it with a path to a pipeline.yaml configuration file. Setting compile=False is an option during initialization.

    from inference import Inference
import os

PATH = os.getcwd()
TAG = "hf"
config_path = f"{PATH}/../checkpoints/{TAG}/pipeline.yaml"
inference = Inference(config_path, compile=False)

  11. Convert GLB meshes to voxel occupancy tensors

    main

    You can convert 3D GLB meshes into voxel occupancy tensors suitable for the Sparse Structure VAE. The process involves:

    1. Loading the GLB file via trimesh.
    2. Rotating the mesh from Y-up to Z-up coordinate system: (x, y, z) → (x, z, -y).
    3. Normalizing vertices to the [-0.5, 0.5]^3 bounding box.
    4. Voxelizing the mesh at a specified resolution.
    5. Filling interior voxels (for watertight meshes) to create a solid occupancy grid.

    Returns a tensor of shape [1, resolution, resolution, resolution] where occupied voxels are set to 1.0.

    def glb_to_voxels(glb_path, resolution=64, save_ply_path=None):
    # ... implementation details ...
    return occupancy, voxel_coords

  12. Transform SAM 3D Objects to OpenGL coordinate system

    main

    SAM 3D Objects are initially generated in a different coordinate system. To make them compatible with the SAM 3D Body (3DB) system, you must transform the posed SAM 3D Objects into the OpenGL coordinate system by flipping the X and Z axes.

    import numpy as np
import open3d as o3d

# Load PLY file
input_path = 'gaussians/human_object_posed.ply'
output_path = 'meshes/human_object/3Dfy_results/0.ply'
mesh = o3d.io.read_point_cloud(input_path)
points = np.asarray(mesh.points)

# Transform to OpenGL coordinate system: flip x and z
points[:, [0, 2]] *= -1
mesh.points = o3d.utility.Vector3dVector(points)
o3d.io.write_point_cloud(output_path, mesh)