RoboSpatial

The repository is organized as follows:

• configs/: YAML configuration files (e.g., example_config.yaml).
• data_loader/: Modules for loading 3D datasets (e.g., embodiedscan_loader.py).
• spatial_analysis/: Core spatial reasoning logic:
  • context/: Spatial context (points relative to an object).
  • configuration/: Spatial configuration (relative position between objects).
  • compatibility/: Spatial compatibility (fitting assessment).
  • grounding.py: 2D object grounding.
  • relationships.py: High-level wrappers for spatial analysis.
  • relationship_utils.py: Geometry and projection utilities.
  • topdown_map.py: 2D top-down occupancy grid functions.
  • obj_properties.py: Object property definitions.
• annotation_generator.py: Orchestrates generation for a single scene.
• run_generation.py: Main script to run generation across datasets/scenes.

Spatial Configuration (spatial_analysis/configuration/configuration.py) determines the pairwise spatial relationship between two objects (obj1 relative to obj2) across camera, world, and object frames (e.g., left/right, infront/behind, above/below, overlapping).

Logic:

• Camera/World-centric: Uses the spatial_configuration_strictness parameter:
  • 'strict': Uses absolute min/max bounds. Requires clear separation; considers objects overlapping if bounds intersect at all.
  • 'lenient': Uses projected centers, average visible depths, and average Z coordinates. More robust to partial occlusion.
• Object-centric: Uses the Separating Axis Theorem (SAT) on 3D OBBs to check for overlap. If not overlapping, determines direction based on obj1's center projected onto obj2's local forward and right axes.

Configuration (data_generation.generation_options):

• spatial_configuration_strictness: Either 'strict' or 'lenient' (default).
• pairwise_relationship_mode:
  • 'unique_categories_only': Only considers pairs where each object is the only instance of its category visible.
  • 'all_visible_objects': Considers all permutations of visible objects.

Spatial Context (spatial_analysis/context/context.py) samples points in the empty space surrounding a reference object and categorizes them by their spatial relationship (infront, behind, left, right) across three frames: objectcentric, cameracentric, and worldcentric.

Logic:

1. Identifies empty space on the floor using a top-down 2D grid.
2. Finds grid points within a context_threshold distance from the reference object's footprint.
3. Projects 3D points onto the image, filtering out points that are off-view or behind the camera.
4. Checks for occlusions using an environment occupancy map.
5. Categorizes non-occluded points based on their position relative to the reference object.
6. Randomly samples up to context_num_samples points per category.

Configuration (data_generation.generation_options):

• context_threshold: Max distance (in world units) from the reference object's footprint to consider sampling.
• context_grid_resolution: Size of cells in the top-down 2D grid (smaller is more precise but slower).
• context_num_samples: Max points to sample per valid category.

Spatial Compatibility (spatial_analysis/compatibility/compatibility.py) assesses if an object (obj_a) can be placed in empty space relative to another object (obj_b) without collision (e.g., left, right, in front, behind, or on_top).

Logic:

1. Samples potential placement points around obj_b using the Spatial Context logic with a dynamic threshold based on object sizes.
2. Simulates placing obj_a at each point.
3. Checks for collisions against the static environment and the reference object obj_b (respecting compatibility_min_distance).
4. A relationship is considered compatible if any sampled point allows the placement.
5. can_fit_on_top is a separate check comparing horizontal dimensions, provided obj_a is placeable and obj_b has a flat surface.

Configuration (data_generation.generation_options):

• compatibility_grid_resolution: Resolution of the 2D grid for environment collision checking.
• compatibility_num_samples: Number of potential placement points to sample.
• compatibility_min_distance: Minimum required distance between obj_a and obj_b in world units.
• compatibility_buffer_ratio: Safety margin applied to obj_a's dimensions (e.g., 0.1 for a 10% buffer).
• context_threshold: The base threshold used for sampling (dynamically increased based on object sizes).

To integrate a custom dataset into the RoboSpatial annotation generation pipeline, you must create a class that inherits from robospatial.data_loader.base_loader.BaseLoader. Your loader is responsible for mapping your dataset's specific format to the interface expected by the pipeline, including scenes, images, and 3D object instances.

Your implementation must provide the following methods:

1. __init__(self, config): Initialize the loader using the data_loading dictionary from your YAML configuration.
2. list_scenes(self, dataset_list): A generator yielding (dataset_name, scene_idx, scene_name) tuples.
3. list_images(self, dataset_name, scene_name): Returns a dictionary of image annotations containing extrinsic (4x4 matrix), intrinsic (4x4 matrix), and img_path.
4. list_objects(self, dataset_name, scene_name, image_ann): Processes visible objects, returning a 5-element tuple containing visible objects, category sets, floor bounds, and all scene objects.

from robospatial.data_loader.base_loader import BaseLoader

class YourCustomLoader(BaseLoader):
    def __init__(self, config):
        super().__init__()
        self.config = config
        # Load annotations using config keys

    def list_scenes(self, dataset_list):
        # yield (dataset_name, scene_idx, scene_name)
        pass

    def list_images(self, dataset_name, scene_name):
        # return {image_id: {'extrinsic': ..., 'intrinsic': ..., 'img_path': ...}}
        pass

    def list_objects(self, dataset_name, scene_name, image_ann):
        # return (vis_objs, unique_vis_categories, multi_vis_categories, floor_bound, all_objs)
        pass

After downloading and preprocessing the EmbodiedScan data, you must update the robospatial/configs/embodiedscan.yaml configuration file to point to your local data directories.

Under the data_loading section, update the following keys:

• image_root: The directory containing extracted images (e.g., scannet/posed_images or 3rscan/<scene_id>/sequence).
• embodiedscan_ann.train: Path to the training .pkl annotation file.
• embodiedscan_ann.val: Path to the validation .pkl annotation file.
• embodiedscan_ann.test: Path to the test .pkl annotation file.

data_loading:
  # ... other settings ...
  image_root: /path/to/your/processed/image/data # <- UPDATE THIS
  embodiedscan_ann:
    train: /path/to/your/EmbodiedScan/data/embodiedscan_infos_train.pkl # <- UPDATE THIS
    val:   /path/to/your/EmbodiedScan/data/embodiedscan_infos_val.pkl   # <- UPDATE THIS
    test:  /path/to/your/EmbodiedScan/data/embodiedscan_infos_test.pkl  # <- UPDATE THIS
  # ... other settings ...