Open3D-ML

You can visualize rich 3D models using Physically Based Rendering (PBR) materials. Material properties are categorized into three types:

• scalar: Uniform values (e.g., material_scalar_metallic).
• vector: Uniform 4-element vectors (e.g., material_vector_base_color).
• texture_map: Spatially varying properties using images (e.g., material_texture_map_albedo).

Requirements:

• To use texture maps, you must provide UV coordinates via vertex_texture_uvs or triangle_texture_uvs.
• For metallic texture maps, you must also provide a base material_scalar_metallic value.
• Specify the shader via the material_name key.

Example dictionary structure:

summary_3d = {
    "vertex_positions": model.vertex["positions"],
    "vertex_normals": model.vertex["normals"],
    "triangle_texture_uvs": model.triangle["texture_uvs"],
    "triangle_indices": model.triangle["indices"],
    "material_name": "defaultLit",
    "material_texture_map_albedo": albedo_image,
    "material_texture_map_normal": normal_image,
    "material_scalar_metallic": 1.0
}

To integrate a completely new dataset type into the library, you can implement a custom class in ml3d/datasets.

1. BaseDataset: Inherit from BaseDataset and implement __init__, get_split, is_tested, and save_test_result.
2. DatasetSplit: Implement a split class (e.g., MyDatasetSplit) that handles the actual data retrieval. It must implement __len__ and get_data(idx), where get_data returns a dictionary containing {'point': points, 'feat': features, 'label': labels}.

from .base_dataset import BaseDataset

class MyDataset(BaseDataset):
    def __init__(self, name="MyDataset"):
        super().__init__(name=name)
        # read file lists.

    def get_split(self, split):
        return MyDatasetSplit(self, split=split)

    def is_tested(self, attr):
        # checks whether attr['name'] is already tested.

    def save_test_result(self, results, attr):
        # save results['predict_labels'] to file.


class MyDatasetSplit():
    def __init__(self, dataset, split='train'):
        self.split = split
        self.path_list = []
        # collect list of files relevant to split.

    def __len__(self):
        return len(self.path_list)

    def get_data(self, idx):
        path = self.path_list[idx]
        points, features, labels = read_pc(path)
        return {'point': points, 'feat': features, 'label': labels}

    def get_attr(self, idx):
        path = self.path_list[idx]
        name = path.split('/')[-1]
        return {'name': name, 'path': path, 'split': self.split}

Inference processes a point cloud and returns results based on a trained model. To perform inference:

1. Load the model using a checkpoint path (ckpt_path).
2. Initialize the pipeline with the model and dataset.
3. Retrieve a specific data sample from a split using split.get_data(index).
4. Call pipeline.run_inference(data).

import open3d.ml.torch as ml3d
from open3d.ml.torch.models import RandLANet
from open3d.ml.torch.pipelines import SemanticSegmentation

# Note: get_module and args are assumed to be defined in the environment
Pipeline = get_module("pipeline", "SemanticSegmentation", "torch")
Model = get_module("model", "RandLANet", "torch")
Dataset = get_module("dataset", "SemanticKITTI")

# Create a checkpoint from the model
RandLANet_model = Model(ckpt_path=args.path_ckpt_randlanet)
SemanticKITTI = Dataset(args.path_semantickitti, use_cache=False)
pipeline = Pipeline(model=RandLANet_model, dataset=SemanticKITTI)

# Get data from the SemanticKITTI dataset using the "train" split
train_split = SemanticKITTI.get_split("train")
data = train_split.get_data(0)

# Run the inference
results = pipeline.run_inference(data)
print(results)

Testing is similar to inference but uses the run_test method. This is typically used to evaluate the model on a predefined test set or a specific data sample.

1. Initialize the pipeline with a checkpointed model and dataset.
2. Retrieve data from a split.
3. Call pipeline.run_test(data).

import open3d.ml.torch as ml3d
from open3d.ml.torch.models import RandLANet
from open3d.ml.torch.pipelines import SemanticSegmentation

Pipeline = get_module("pipeline", "SemanticSegmentation", "torch")
Model = get_module("model", "RandLANet", "torch")
Dataset = get_module("dataset", "SemanticKITTI")

# Create a checkpoint
RandLANet_model = Model(ckpt_path=args.path_ckpt_randlanet)
SemanticKITTI = Dataset(args.path_semantickitti, use_cache=False)
pipeline = Pipeline(model=RandLANet_model, dataset=SemanticKITTI)

# Get data from the SemanticKITTI dataset using the "train" split
train_split = SemanticKITTI.get_split("train")
data = train_split.get_data(0)

# Run the test
pipeline.run_test(data)

To avoid writing redundant geometry data (like vertex positions or normals) when only colors or other properties change, you can pass an integer (the step reference) instead of the full tensor for those properties. This significantly reduces the size of the summary files.

for step in range(3):
    cube.paint_uniform_color(colors[step])
    cube_summary = to_dict_batch([cube])
    if step > 0:
        # Use 0 to indicate reuse of geometry from the previous step
        cube_summary['vertex_positions'] = 0
        cube_summary['vertex_normals'] = 0
    writer.add_3d('cube', cube_summary, step=step)

for step in range(3):
        cube.paint_uniform_color(colors[step])
        cube_summary = to_dict_batch([cube])
        if step > 0:
            cube_summary['vertex_positions'] = 0
            cube_summary['vertex_normals'] = 0
        writer.add_3d('cube', cube_summary, step=step)
        cylinder.paint_uniform_color(colors[step])
        cylinder_summary = to_dict_batch([cylinder])
        if step > 0:
            cylinder_summary['vertex_positions'] = 0
            cylinder_summary['vertex_normals'] = 0
        writer.add_3d('cylinder', cylinder_summary, step=step)