meshgpt-pytorch

You can enable text conditioning in MeshTransformer by setting condition_on_text = True.

Training with text: Pass a list of descriptions to the texts keyword argument during the forward pass.

Generating with text: Pass descriptions to .generate(). You can use cond_scale to enable classifier-free guidance (typically between 3. and 10.). You can also set remove_parallel_component = True as per recent research.

transformer = MeshTransformer(
    autoencoder,
    dim = 512,
    max_seq_len = 768,
    condition_on_text = True
)

# Training with text
loss = transformer(
    vertices = vertices,
    faces = faces,
    texts = ['a high chair', 'a small teapot'],
)
loss.backward()

# Generating with text and classifier-free guidance
faces_coordinates, face_mask = transformer.generate(
    texts = ['a long table'],
    cond_scale = 8.,
    remove_parallel_component = True
)

The MeshAutoencoder is used to encode and reconstruct 3D meshes. It takes vertices and faces as input.

Important: For variable-length meshes, ensure that faces are padded with -1.

Input Shapes:

• vertices: (batch, num_vertices, 3)
• faces: (batch, num_faces, 3)

To convert meshes into tokens for use in multimodal transformers, use the .tokenize method.

import torch
from meshgpt_pytorch import MeshAutoencoder

autoencoder = MeshAutoencoder(num_discrete_coors = 128)

vertices = torch.randn((2, 121, 3))            # (batch, num vertices, coor (3))
faces = torch.randint(0, 121, (2, 64, 3))      # (batch, num faces, vertices (3))

# Ensure faces are padded with -1 for variable length if necessary

loss = autoencoder(vertices = vertices, faces = faces)

# Tokenizing for multimodal use
mesh_token_ids = autoencoder.tokenize(
    vertices = vertices,
    faces = faces
)
# mesh_token_ids shape: (batch, num face vertices, residual quantized layer)

The MeshTransformer models the sequence of face vertices after an autoencoder has been trained. It can be used for training on raw face data and for sampling novel 3D assets.

Basic Training/Forward Pass: Pass vertices and faces to the transformer.

Sampling/Generation: Call .generate() to sample new assets. It returns faces_coordinates and face_mask.

• faces_coordinates: (batch, num faces, vertices (3), coordinates (3))
• face_mask: (batch, num faces)

import torch
from meshgpt_pytorch import MeshAutoencoder, MeshTransformer

autoencoder = MeshAutoencoder(num_discrete_coors = 128)
vertices = torch.randn((2, 121, 3))
faces = torch.randint(0, 121, (2, 64, 3))

transformer = MeshTransformer(
    autoencoder,
    dim = 512,
    max_seq_len = 768
)

loss = transformer(vertices = vertices, faces = faces)
loss.backward()

# Sampling
faces_coordinates, face_mask = transformer.generate()