DOCSEARCHbyAlgolia
Submit Docs

VLFM (Vision-Language Frontier Maps)

repository·main·Indexed 20 days ago

https://github.com/rai-opensource/vlfm

A zero-shot semantic navigation approach using vision-language models to navigate toward unseen semantic objects in novel environments. VLFM builds occupancy maps to identify frontiers and utilizes language-grounded value maps to select exploration targets. Version 0.1 supports simulation experiments via Habitat and deployment on the Spot robot.

Tokens
1.6K
Snippets
5
Records
6
Agent score
23%

What's inside vlfm

  1. Download the HM3D dataset

    main

    To use the HM3D dataset, you must first set your Matterport credentials and data directory as environment variables. Then, use habitat_sim utilities to download the 3D scans (scenes) and manually download/unzip the ObjectNav episodes into the correct directory structure under your DATA_DIR.

    # Set environment variables
MATTERPORT_TOKEN_ID=<YOUR_ID>
MATTERPORT_TOKEN_SECRET=<YOUR_SECRET>
DATA_DIR=</path/to/vlfm/data>
HM3D_OBJECTNAV=https://dl.fbaipublicfiles.com/habitat/data/datasets/objectnav/hm3d/v1/objectnav_hm3d_v1.zip

# Download HM3D 3D scans
python -m habitat_sim.utils.datasets_download \
  --username $MATTERPORT_TOKEN_ID --password $MATTERPORT_TOKEN_SECRET \
  --uids hm3d_train_v0.2 \
  --data-path $DATA_DIR &&
python -m habitat_sim.utils.datasets_download \
  --username $MATTERPORT_TOKEN_ID --password $MATTERPORT_TOKEN_SECRET \
  --uids hm3d_val_v0.2 \
  --data-path $DATA_DIR &&

# Download and organize HM3D ObjectNav episodes
wget $HM3D_OBJECTNAV &&
unzip objectnav_hm3d_v1.zip &&
mkdir -p $DATA_DIR/datasets/objectnav/hm3d  &&
mv objectnav_hm3d_v1 $DATA_DIR/datasets/objectnav/hm3d/v1 &&
rm objectnav_hm3d_v1.zip

  2. Evaluate VLFM in Habitat

    main

    Before running evaluation, you must launch the VLM servers to load models into memory via flask. This is done using a provided script that creates a tmux session. Once the servers are running, you can execute the evaluation module for either the HM3D or MP3D datasets.

    # 1. Launch VLM servers (run once, creates a tmux session)
chmod +x ./scripts/launch_vlm_servers.sh
./scripts/launch_vlm_servers.sh

# 2. Run evaluation on HM3D
python -m vlfm.run

# 3. Run evaluation on MP3D
python -m vlfm.run habitat.dataset.data_path=data/datasets/objectnav/mp3d/val/val.json.gz

  3. Install VLFM

    main

    To install VLFM, first create and activate a Python 3.9 conda environment, then install the required PyTorch and vision-language model dependencies. Depending on your target hardware, install the project in editable mode with either the [habitat] extra for simulation experiments or the [reality] extra for deployment on the Spot robot. Additionally, you must clone the yolov7 repository into the VLFM directory.

    # Create environment
conda_env_name=vlfm
conda create -n $conda_env_name python=3.9 -y
conda activate $conda_env_name

# Install core dependencies
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.pytorch.org/whl/torch_stable.html
pip install git+https://github.com/IDEA-Research/GroundingDINO.git@eeba084341aaa454ce13cb32fa7fd9282fc73a67 salesforce-lavis==1.0.2

# Install VLFM (choose one)
pip install -e .[habitat]  # For simulation
pip install -e .[reality]  # For Spot robot

# Clone yolov7 dependency
git clone git@github.com:WongKinYiu/yolov7.git

  4. Download required model weights

    main

    VLFM requires several pre-trained weights to be stored in the data/ directory. Ensure the following files are downloaded and placed correctly:

    • mobile_sam.pt (from MobileSAM)
    • groundingdino_swint_ogc.pth (from GroundingDINO)
    • yolov7-e6e.pt (from yolov7)
    • pointnav_weights.pth (included in the repository's data subdirectory)

  5. Run VLFM experiments via the main entrypoint

    main

    The vlfm.run module provides the main entrypoint for executing VLFM (Vision-Language Foundation Model) experiments using Hydra and Habitat. The script is designed to be run in an environment where habitat is installed.

    Before running the script, ensure that:

    1. A data/ directory exists in your working directory.
    2. Dummy policy weights are available at data/dummy_policy.pth. If they are missing, generate them using:
      python -m vlfm.utils.generate_dummy_policy

    By default, the script uses the experiments/vlfm_objectnav_hm3d configuration from the ../config directory. It automatically determines whether to run in train or eval mode based on the cfg.habitat_baselines.evaluate flag.

    python -m vlfm.run

  6. HabitatConfigPlugin for Hydra configuration discovery

    main

    The HabitatConfigPlugin is a custom Hydra SearchPathPlugin used to extend the configuration search path. It ensures that Habitat-specific configurations located in the config/ directory are discoverable by the Hydra engine during experiment execution.

    class HabitatConfigPlugin(SearchPathPlugin):
    def manipulate_search_path(self, search_path: ConfigSearchPath) -> None:
        search_path.append(provider="habitat", path="config/")