ConceptGraphs

To set up the ConceptGraphs environment, it is recommended to use a virtual environment (Conda or virtualenv) with Python 3.10.12. While an environment.yml is provided, following the manual installation steps below is recommended to avoid dependency conflicts.

Key dependencies include tyro, open_clip_torch, wandb, h5py, openai, hydra-core, distinctipy, and ultralytics (for YOLO). You will also need faiss-cpu and a specific version of pytorch3d compatible with your CUDA setup.

conda create -n conceptgraph anaconda python=3.10
conda activate conceptgraph

# Install the required libraries
pip install tyro open_clip_torch wandb h5py openai hydra-core distinctipy

# for yolo
pip install ultralytics

# Install the Faiss library (CPU version should be fine)
conda install -c pytorch faiss-cpu=1.7.4 mkl=2021 blas=1.0=mkl

##### Install Pytorch according to your own setup #####
# For example, if you have a GPU with CUDA 11.8 (We tested it Pytorch 2.0.1)
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia

# Install Pytorch3D
conda install https://anaconda.org/pytorch3d/pytorch3d/0.7.4/download/linux-64/pytorch3d-0.7.4-py310_cu118_pyt201.tar.bz2

Builds an object-based 3D map using the previously extracted segmentation results.

Key Options:

• save_objects_all_frames=True: Saves mapping results at every frame, enabling animated visualization via scripts/animate_mapping_interactive.py or scripts/animate_mapping_save.py.
• merge_interval, merge_visual_sim_thresh, merge_text_sim_thresh: Used to perform overlap-based merging during mapping.

Results are saved in $REPLICA_ROOT/$SCENE_NAME/pcd_saves. It is recommended to use the .pkl.gz files with the _post suffix (post-processed results).

# Example: ConceptGraphs-Detect mapping
SCENE_NAME=room0
THRESHOLD=1.2
python slam/cfslam_pipeline_batch.py \
    dataset_root=$REPLICA_ROOT \
    dataset_config=$REPLICA_CONFIG_PATH \
    stride=5 \
    scene_id=$SCENE_NAME \
    spatial_sim_type=overlap \
    mask_conf_threshold=0.25 \
    match_method=sim_sum \
    sim_threshold=${THRESHOLD} \
    dbscan_eps=0.1 \
    gsa_variant=ram_withbg_allclasses \
    skip_bg=False \
    max_bbox_area_ratio=0.5 \
    save_suffix=overlap_maskconf0.25_simsum${THRESHOLD}_dbscan.1

To generate datasets for AI2Thor experiments, you must first set up a specific fork of ai2thor.

Workflow:

1. Grid Map Generation: Create a densely captured grid map using --sample_method uniform.
2. Human-controlled Trajectory: Use --interact to open a Unity window and record a trajectory via keyboard/GUI.
3. Observation Generation: Generate observations from the saved trajectory using --sample_method from_file.
4. Randomized Trajectory: Generate trajectories with randomly moved objects using --randomize_move_moveable_ratio and --randomize_move_pickupable_ratio.

# 1. Generate Grid Map
python scripts/generate_ai2thor_dataset.py --dataset_root $AI2THOR_DATASET_ROOT --scene_name $SCENE_NAME --sample_method uniform --n_sample -1 --grid_size 0.5

# 2. Generate Human Trajectory
python scripts/generate_ai2thor_dataset.py --dataset_root $AI2THOR_DATASET_ROOT --scene_name $SCENE_NAME --interact

# 3. Generate Observations from file
python scripts/generate_ai2thor_dataset.py --dataset_root $AI2THOR_DATASET_ROOT --scene_name $SCENE_NAME --sample_method from_file

The ./scripts/streamlined_detections.py script uses the hydra package for configuration management. Instead of passing numerous command-line arguments, you should modify the .yaml files located in ./conceptgraph/hydra_configs/.

Configuration Hierarchy

Configuration values are loaded in a specific order, where later files overwrite values from earlier ones:

1. base.yaml (the foundation)
2. replica.yaml (overwrites base)
3. sam.yaml (overwrites previous)
4. _self_ (the values defined in your specific config file, e.g., streamlined_detections.yaml, which take highest priority).

Crucial Step: You must edit the path values in replica.yaml to point to your specific dataset.

You can also create and add your own custom .yaml files to ./conceptgraph/hydra_configs/ to extend the configuration.

# Example: ./conceptgraph/hydra_configs/streamlined_detections.yaml
defaults:
  - base
  - replica
  - sam
  - _self_

stride: 1
exp_suffix: _streamlined_yolo_stride50_no_bg24
save_video: true

This step extracts (Detection) Segmentation results and per-region CLIP features. There are two modes:

1. ConceptGraphs (Class-agnostic): Uses SAM in "segment all" mode to extract class-agnostic masks. Set --class_set none.
2. ConceptGraphs-Detect (Class-aware): Uses a tagging and detection model to extract class-aware bounding boxes, which then prompt SAM for segmentation. Set --class_set to a value like ram.

Results and visualizations are saved to $REPLICA_ROOT/$SCENE_NAME/.

SCENE_NAME=room0

# Option 1: ConceptGraphs (without open-vocab detector)
python scripts/generate_gsa_results.py \
    --dataset_root $REPLICA_ROOT \
    --dataset_config $REPLICA_CONFIG_PATH \
    --scene_id $SCENE_NAME \
    --class_set none \
    --stride 5

# Option 2: ConceptGraphs-Detect
CLASS_SET=ram
python scripts/generate_gsa_results.py \
    --dataset_root $REPLICA_ROOT \
    --dataset_config $REPLICA_CONFIG_PATH \
    --scene_id $SCENE_NAME \
    --class_set $CLASS_SET \
    --box_threshold 0.2 \
    --text_threshold 0.2 \
    --stride 5 \
    --add_bg_classes \
    --accumu_classes \
    --exp_suffix withbg_allclasses