3D-Mem: 3D Scene Memory Framework

To use 3D-Mem, set up a Python 3.9 environment on Linux using Conda and install the required dependencies including PyTorch, Habitat-Sim, PyTorch3D, and several vision/ML libraries.

Follow these steps to create the environment and install the packages:

# Create and activate the conda environment
conda create -n 3dmem python=3.9 -y && conda activate 3dmem

# Install PyTorch with CUDA 11.8 support
pip install torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/cu118

# Install Habitat-Sim and Faiss
conda install -c conda-forge -c aihabitat habitat-sim=0.2.5 headless faiss-cpu=1.7.4 -y

# Install PyTorch3D
conda install https://anaconda.org/pytorch3d/pytorch3d/0.7.4/download/linux-64/pytorch3d-0.7.4-py39_cu118_pyt201.tar.bz2 -y

# Install remaining dependencies
pip install omegaconf==2.3.0 open-clip-torch==2.26.1 ultralytics==8.2.31 supervision==0.21.0 opencv-python-headless==4.10.* \
 scikit-learn==1.4 scikit-image==0.22 open3d==0.18.0 hipart==1.0.4 openai==1.35.3 httpx==0.27.2

These functions help determine optimal camera positions to observe objects or groups of objects in a scene, ensuring the camera is placed in unoccupied (navigable) space and has a clear line of sight.

get_proper_snapshot_observation_point

Finds a single optimal observation point for a set of object centers. It uses the structure tensor of the object centers to determine the 'long axis' and 'short axis' of the group, then attempts to place the camera opposite the short axis to capture the group effectively.

get_random_snapshot_observation_point

Similar to the above, but selects an observation point randomly from valid unoccupied candidates, weighted by their position relative to the object group's orientation.

Common Parameters:

• obj_centers: [N, 2] array of object centers in voxel space.
• unoccupied_map: [H, W] boolean array where True indicates navigable space.
• min_obs_dist / max_obs_dist: Distance range from the object center to the camera.
• snapshot_observation_point: A fallback/initial point for the observation.

Returns: A [2] array representing the optimal (or fallback) observation point in voxel coordinates.

The TSDFPlanner uses several dataclasses to manage exploration and semantic data:

• Frontier: Represents an unexplored region.

  • position: np.ndarray (voxel grid coordinates).
  • orientation: np.ndarray (directional vector).
  • region: np.ndarray (boolean mask of the voxel grid).
  • frontier_id: int.
  • image: str (path to snapshot image).
  • target_detected: bool.
  • feature: torch.Tensor (image features).

• SnapShot: Represents a semantic observation of a cluster of objects.

  • image: str (path to image).
  • color: Tuple[float, float, float].
  • obs_point: np.ndarray (voxel grid position).
  • full_obj_list: Dict[int, float] (object ID to confidence).
  • cluster: List[int] (list of object IDs).
  • position: np.ndarray (calculated center).

• SceneGraphItem: Represents a detected object in the scene.

  • object_id: int.
  • bbox_center: np.ndarray.
  • confidence: float.
  • image: str.

The OptionalWandB class is a singleton wrapper designed to allow your code to use wandb methods (like .init(), .log(), and .finish()) without requiring the wandb library to be installed or enabled.

If wandb is installed and enabled via configuration, OptionalWandB acts as a proxy, forwarding calls to the actual wandb library. If wandb is missing or disabled, the calls are silently ignored (or logged as debug messages), preventing ImportError or runtime crashes.

Implementation Pattern

1. In your main script: Instantiate the singleton and configure it using .set_use_wandb(bool) based on your project settings.
2. In utility or secondary scripts: Instantiate the singleton to get the same instance, but do not call .set_use_wandb() again; it will inherit the state set by the main script.

from custom_wandb import OptionalWandB

# 1. Setup in main script
owandb = OptionalWandB()
owandb.set_use_wandb(True)  # or cfg.use_wandb

# 2. Use like normal wandb
owandb.init(project="my_project")
owandb.log({"metric": 42})
owandb.finish()

Use the following configuration keys to control output and rendering:

• save_objects_all_frames: If True, saves the state of objects for every frame in a folder named objects_all_frames within the scene directory.
• save_pcd: If True, saves the full point cloud as a .pkl.gz file. It saves both the raw and the post-processed versions.
• vis_render: If True, enables online object rendering for visualization.
• render_camera_path: Path to the pinhole camera parameters for rendering.
• debug_render: If True, opens an interactive Open3D visualization window during the process.
• class_agnostic: If True, colors objects by instance; otherwise, colors them by their most common classes.
• obj_min_detections: Minimum number of detections required for an object to be included in saved visualizations/outputs.
• skip_bg: If True, skips background detection (e.g., wall, floor, ceiling).

Updates the existing map by either adding new detections as new objects or merging detections into existing objects based on match indices.

Args:

• detection_list: DetectionList of new detections.
• objects: MapObjectList of current map objects.
• match_indices: A list where each entry is the index of the existing object a detection matches with, or None if it is a new object.
• downsample_voxel_size: Voxel size for processing.
• dbscan_remove_noise: Boolean to enable/disable DBSCAN noise removal.
• dbscan_eps: DBSCAN epsilon parameter.
• dbscan_min_points: DBSCAN minimum points parameter.
• spatial_sim_type: The type of spatial similarity to use during merging.
• device: The compute device (e.g., 'cpu' or 'cuda').

Returns an updated MapObjectList.

updated_objects = merge_obj_matches(
    detection_list=detections,
    objects=current_objects,
    match_indices=matches,
    downsample_voxel_size=0.05,
    dbscan_remove_noise=True,
    dbscan_eps=0.02,
    dbscan_min_points=10,
    spatial_sim_type="iou",
    device="cuda"
)