FoundationStereo Documentation

Once inside the container, install the required ONNX runtime, PyCUDA, and ONNX-TensorRT packages. Note that a specific wheel is required for onnxruntime-gpu on aarch64 architectures.

# Install ONNX runtime for aarch64
pip install https://github.com/ultralytics/assets/releases/download/v0.0.0/onnxruntime_gpu-1.20.0-cp310-cp310-linux_aarch64.whl

pip3 install pycuda --user

# Install ONNX-TensorRT
git clone https://github.com/onnx/onnx-tensorrt.git
cd onnx-tensorrt
python3 setup.py install

# Install specific numpy version
pip install numpy==1.23.5

# Update PYTHONPATH
export PYTHONPATH=$PYTHONPATH:<path_to_onnx-tensorrt>

Launch the Docker container with necessary permissions for X11 forwarding, GPU access, and hardware acceleration. Ensure you initialize X11 forwarding with xhost + before running the container.

# Initialize X11 forwarding
xhost +

# Run the Docker container
docker run \
--privileged \
--rm \
-it \
--network host \
--ipc host \
--gpus all \
--runtime=nvidia \
--name=foundation_stereo_container \
-e DISPLAY \
-e NVIDIA_DISABLE_REQUIRE=1 \
-e ISAAC_ROS_DEV_DIR=/workspace/isaac_ros-dev \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v /tmp/.X11-unix:/tmp/.X11-unix:cached \
-e DISPLAY="${DISPLAY}" \
isaac_ros_dev-aarch64:latest

Install the environment using Conda. Note that flash-attn must be installed separately after environment creation to avoid errors.

conda env create -f environment.yml
conda run -n foundation_stereo pip install flash-attn
conda activate foundation_stereo

You can run Channel-Adaptive DINO training using a SLURM cluster environment with submitit. This example uses 4 A100-80GB nodes (32 GPUs) and targets the HPA-FoV dataset.

Note: Ensure the dinov2 package is in your Python module search path by prefixing the command with PYTHONPATH=..

python dinov2/run/train/train.py \
    --nodes 4 \
    --config-file dinov2/configs/train/hpafov_vitl16_boc.yaml \
    --output-dir <PATH/TO/OUTPUT/DIR> \
    train.dataset_path=HPAFoV:split=LARGE_REPRODUCE:root=<PATH/TO/DATASET>:wildcard=SEPARATE_CHANNELS"

Download the desired foundation model and place the entire folder (e.g., 23-51-11) under the ./pretrained_models/ directory.

Available models:

• 23-51-11: Best performing model for general use (Vit-large).
• 11-33-40: Faster inference with slightly lower accuracy (Vit-small).
• NVIDIA-TAO: Commercial version (adapted from Vit-small).

The ImageNet implementation requires specific metadata files (e.g., class-ids-TRAIN.npy, entries-TEST.npy) located in an extra directory. You can generate these files using the following Python snippet:

from dinov2.data.datasets import ImageNet

for split in ImageNet.Split:
    dataset = ImageNet(split=split, root="<ROOT>", extra="<EXTRA>")
    dataset.dump_extra()

Note: <ROOT> is your dataset root and <EXTRA> is your metadata directory. These can be the same directory.

DINOv2 models act as vision backbones and can be used for various tasks without fine-tuning by applying simple heads to the extracted features:

• Depth Estimation & Semantic Segmentation: Use linear layers on patch tokens.
• Image Classification:
  • Use k-NN classifiers on the class token.
  • Use logistic regression classifiers on the class token.
  • Use a linear layer on the class token combined with the average of the patch tokens.
• Image Retrieval: Use nearest neighbors on the features.

To deploy FoundationStereo on Jetson devices, you must first set up the NVIDIA Isaac ROS developer environment. This involves cloning the isaac_ros_common repository into your Isaac ROS workspace source directory.

1. Follow the NVIDIA Isaac ROS Developer Environment Setup guide.
2. Clone isaac_ros_common using the release-3.2 branch.

cd ${ISAAC_ROS_WS}/src && \
git clone -b release-3.2 https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common.git isaac_ros_common

# download the image (running the container might fail; just ignore, we are using our own run script)
cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
./scripts/run_dev.sh

To generate a point cloud for your own data, you must specify the camera intrinsics. In the intrinsic file provided via arguments:

1. The first line should be the flattened 1x9 intrinsic matrix.
2. The second line should be the baseline (distance) between the left and right cameras in meters.