DOCSEARCHbyAlgolia
Submit Docs

PyTorch3D Documentation

repository·main·Indexed 27 days ago

https://github.com/facebookresearch/pytorch3d

A library of efficient, differentiable, and GPU-accelerated components for 3D Computer Vision research, specifically designed to work with PyTorch. The documentation covers installation via source, wheels, or Google Colab, building NVIDIA CUB conda packages, and detailed guides for the Implicitron trainer project, including training, evaluation, and custom plugin implementation.

Tokens
26.8K
Snippets
70
Records
136
Agent score
94%

What's inside PyTorch3D

  1. Overview of PyTorch3D features

    main

    PyTorch3D is a library for 3D Computer Vision research built on PyTorch. It provides efficient, differentiable, and GPU-accelerated components designed to integrate with deep learning methods.

    Key capabilities include:

    • Triangle Mesh Manipulation: Data structures for storing and performing operations like projective transformations, graph convolution, sampling, and loss functions on meshes.
    • Differentiable Rendering: A mesh renderer that supports backpropagation.
    • Implicit Representations: Includes Implicitron, a framework for new-view synthesis via implicit representations.

    All operators are implemented using PyTorch tensors and can handle minibatches of heterogeneous data.

  2. Overview of PyTorch3D Differentiable Rendering

    main

    PyTorch3D provides a modular, differentiable renderer designed for computer vision research. Unlike many existing implementations that bundle components into large, monolithic CUDA kernels, PyTorch3D decouples the rasterization and shading steps.

    Key features include:

    • Modular Design: The core rasterization step is optimized in CUDA, while the rest of the pipeline is implemented in pure PyTorch, making it easy to customize and extend.
    • Heterogeneous Batching: Supports batches where meshes may have different numbers of vertices and faces.
    • Multi-backend Support: Parallel implementations are available in PyTorch, C++, and CUDA.
    • Differentiability: Enables relating 2D image pixels back to 3D scene properties (like mesh vertex positions) via backpropagation.

  3. Use pytorch3d.implicitron.tools modules

    main

    The pytorch3d.implicitron.tools package provides a collection of utility modules designed to support workflows within the Implicitron framework. These tools cover various aspects of implicit reconstruction, including camera handling, data I/O, and visualization.

    Available utility modules include:

    • camera_utils: Utilities for camera parameter manipulation and handling.
    • circle_fitting: Tools for circle fitting operations.
    • config: Configuration management utilities.
    • eval_video_trajectory: Tools for evaluating video trajectories.
    • image_utils: Image processing and manipulation utilities.
    • metric_utils: Utilities for calculating various metrics.
    • model_io: Input/Output operations for models.
    • point_cloud_utils: Utilities for working with point cloud data.
    • rasterize_mc: Monte Carlo rasterization tools.
    • stats: Statistical calculation utilities.
    • video_writer: Tools for writing video files.
    • vis_utils: Visualization utilities.

  4. Understand the Implicitron Experiment structure

    main

    The Experiment object is the main driver for the trainer loop and consists of four top-level replaceable components. The configuration structure is automatically parsed from the module hierarchy.

    • data_source: A DataSourceBase (defaults to ImplicitronDataSource) that constructs datasets and dataloaders.
    • model_factory: A ModelFactoryBase (defaults to ImplicitronModelFactory) that constructs the model (e.g., OverfitModel or GenericModel).
    • optimizer_factory: An OptimizerFactoryBase (defaults to ImplicitronOptimizerFactory) that constructs the optimizer.
    • training_loop: A TrainingLoopBase (defaults to ImplicitronTrainingLoop) that defines the main training loop.

    Configuration Hierarchy Examples:

    • For ImplicitronModelFactory with a generic model: parameters are under model_factory_ImplicitronModelFactory_args.model_GenericModel_args.
    • For dataset parameters: parameters are under data_source_ImplicitronDataSource_args.

  6. Understand PyTorch3D Camera Coordinate Systems

    main

    PyTorch3D utilizes four primary coordinate systems for 3D data processing:

    • World coordinate system: The global system where the scene/objects reside.
    • Camera view coordinate system: Origin is on the image plane. In PyTorch3D, +X points left, +Y points up, and +Z points out from the image plane (towards the scene). Transformation from world to view uses rotation R and translation T.
    • NDC (Normalized Device Coordinate) coordinate system: The view volume. For square images, the volume ranges from (+1, +1, znear) (top left near) to (-1, -1, zfar) (bottom right far). For non-square images, the smaller dimension ranges [-1, 1] and the larger dimension ranges [-s, s] where s is the aspect ratio.
    • Screen coordinate system: Pixel-based representation where (0,0) is the top-left pixel and (W,H) is the bottom-right pixel.

    Crucial for Rendering: The PyTorch3D renderer (meshes and point clouds) assumes input points are in NDC space. Ensure your data and cameras follow the +X:left, +Y:up, and +Z:from us to scene convention before rendering.

  7. Install PyTorch3D Core Library Dependencies

    main

    Before installing PyTorch3D, ensure you have the required runtime dependencies. For a standard Conda setup, use the following commands to create an environment and install PyTorch, torchvision, and iopath:

    conda create -n pytorch3d python=3.9
conda activate pytorch3d
conda install pytorch=1.13.0 torchvision pytorch-cuda=11.6 -c pytorch -c nvidia
conda install -c iopath iopath

  8. Customize shaders with texturing, lighting, and blending

    main

    Shaders in PyTorch3D are highly flexible and can combine several rendering steps:

    • Texturing: Interpolating vertex RGB, vertex UV coordinates (sampling from a texture map), or using a Texture Atlas.
    • Lighting/Shading: Implementing models like Ambient, Diffuse, Specular, Phong, Gouraud, or Flat shading.
    • Blending:
      • Hard blending: Uses only the closest face for each pixel.
      • Soft blending: Uses a weighted sum of the top K faces per pixel.

    Commonly available shader combinations include HardPhongShader, SoftPhongShader, HardGouraudShader, SoftGouraudShader, HardFlatShader, and SoftSilhouetteShader.

  9. Install prebuilt PyTorch3D binaries via Anaconda Cloud (Linux only)

    main

    On Linux, you can install prebuilt binaries with CUDA support directly from Anaconda Cloud.

    Official Build:

    conda install pytorch3d -c pytorch3d

    Nightly (Alpha) Build:

    conda install pytorch3d -c pytorch3d-nightly

  10. PyTorch3D Tutorials

    main

    PyTorch3D provides several tutorial notebooks to help you get started with common 3D vision tasks. Key tutorials include:

  11. Batch heterogeneous meshes using the Meshes data structure

    main

    When working with meshes of different sizes (different number of vertices or faces), the Meshes data structure provides three modes to handle batching. Given a batch meshes = Meshes(verts = [v1, v2], faces = [f1, f2]) where v1, v2 are vertex tensors and f1, f2 are face tensors:

    1. List Mode: Returns the examples as a list of individual tensors. Use this when you need to iterate over meshes individually.

      • meshes.verts_list(): Returns [v1, v2]
      • meshes.faces_list(): Returns [f1, f2]

    2. Padded Mode: Constructs a single tensor by padding extra values to match the largest mesh in the batch. Use this for operators that require fixed-size tensor inputs.

      • meshes.verts_padded(): Returns a tensor of shape N x max(V) x 3, padding extra vertices with 0s.
      • meshes.faces_padded(): Returns a tensor of shape N x max(F) x 3, padding extra faces with -1s.

    3. Packed Mode: Concatenates all meshes in the batch into a single large tensor. This is often the most efficient mode for optimized PyTorch operations.

      • meshes.verts_packed(): Returns a tensor of shape (sum of all V) x 3.
      • meshes.faces_packed(): Returns a tensor of shape (sum of all F) x 3.

    PyTorch3D allows for efficient conversion between these modes to support different network layers (e.g., switching from a padded input for vertex alignment to a packed input for graph convolutions).

  12. Install PyTorch3D from GitHub source

    main

    You can install PyTorch3D directly from the GitHub repository using pip. CUDA support is included if CUDA is available in PyTorch or if the FORCE_CUDA=1 environment variable is set.

    Main Branch:

    pip install "git+https://github.com/facebookresearch/pytorch3d.git"

    Stable Release:

    pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"

    macOS Specific Installation: On macOS, you must provide specific deployment target and compiler environment variables:

    MACOSX_DEPLOYMENT_TARGET=10.14 CC=clang CXX=clang++ pip install "git+https://github.com/facebookresearch/pytorch3d.git"
    pip install "git+https://github.com/facebookresearch/pytorch3d.git"
# Or for stable
pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"