DOCSEARCHbyAlgolia
Submit Docs

ARKitScenes Documentation

repository·main·Indexed 21 days ago

https://github.com/apple/arkitscenes

A large-scale, real-world RGB-D dataset captured with mobile Apple LiDAR for 3D indoor scene understanding. It includes 5,047 captures across 1,661 unique scenes, featuring high-resolution ground truth, 3D annotations for furniture, and camera intrinsic matrices. The repository provides tools and training code for two primary tasks: 3D Object Detection (3DOD) and RGB-D guided depth upsampling, including scripts for data preparation, visualization, and downloading specific dataset subsets (RAW, 3DOD, and upsampling).

Tokens
3.9K
Snippets
10
Records
21
Agent score
74%

What's inside ARKitScenes

  1. Overview of ARKitScenes

    main

    ARKitScenes is a large-scale RGB-D dataset designed for 3D indoor scene understanding, captured using mobile Apple LiDAR scanners. It contains 5,047 captures across 1,661 unique scenes.

    Key assets provided include:

    • Raw and Processed Data: Camera poses and surface reconstructions for each scene.
    • High-Resolution Ground Truth: High-resolution depth maps (captured via stationary laser scanner) and registered RGB-D frames for a subset of 2,257 captures (841 unique scenes).
    • 3D Annotations: Manually labeled 3D oriented bounding boxes for a large taxonomy of furniture.
    • Task-Specific Code: Scripts and training code for 3D Object Detection (3DOD) and RGB-D guided depth upsampling.

  2. Identify ARKitScenes dataset types and contents

    main

    ARKitScenes consists of three primary datasets:

    1. 3dod: Used for 3D object detection training. Contains low-resolution RGB images, low-resolution depth images, and labels. (Total size: ~623.4 GB for 5047 scans).
    2. upsampling: Used for depth upsampling training. Contains high-resolution RGB images, low-resolution depth images, and high-resolution depth images.
    3. raw: The complete dataset containing all available ARKitScenes data, including all assets from 3dod and upsampling plus additional assets.

  3. Relationship between RAW, 3DOD, and Depth Upsampling datasets

    main

    The RAW dataset is the superset of all available ARKitScenes data. Other datasets are subsets of it:

    • 3DOD dataset: A subset of the 60FPS category.
    • Depth upsampling dataset: A subset of the 60FPS and Filtered 10FPS categories.

    Note on Compatibility: Because the RAW and 3DOD datasets were created at different times, you may encounter slight differences in the lowres_depth and confidence assets when comparing them.

  4. Organize and format depth upsampling input data

    main

    The depth upsampling dataset is organized into Training and Validation bins. Each video ID contains several asset folders. The input data consists of the following components:

    • color: RGB images (1920x1440).
    • highres_depth: Ground-truth depth images projected from Faro laser scanner meshes (1920x1440).
    • lowres_depth: Depth images acquired by AppleDepth Lidar (256x192).
    • confidence: Confidence maps for the AppleDepth depth images (256x192).
    • metadata.csv: Metadata per video (e.g., sky direction: up/down/left/right).
    • val_attributes.csv: Attributes per sample (e.g., transparent_or_reflective), used only for the Validation bin.

    Directory Structure Example:

    ARKitScenes/depth_upsampling/
├── Training
│   └── [video_id]
│       ├── color
│       ├── highres_depth
│       ├── lowres_depth
│       └── confidence
└── Validation
    └── ...

  5. Understand the RAW dataset asset categories

    main

    The RAW dataset contains various assets sampled at different rates. These are organized into five main categories:

    1. 60FPS: Low-resolution synchronized assets (256x192).
      • lowres_wide: RGB images from the wide camera.
      • lowres_depth: Depth images from AppleDepth Lidar.
      • confidence: Confidence maps for the lowres_depth images.
    2. Filtered 10FPS: High-resolution synchronized assets (1920x1440) filtered for high error between highres_depth and lowres_depth.
      • highres_depth: Ground-truth depth projected from Faro laser scanner meshes.
      • wide: RGB images from the wide camera.
    3. 10FPS: VGA resolution assets (640x480) from the ultra wide camera (note: these have different timestamps than Category 2).
      • ultrawide: RGB images from the ultra wide camera.
    4. 30FPS: VGA resolution assets (640x480) from the wide camera.
      • vga_wide: RGB images from the wide camera.
    5. Per venue: High-resolution laser scanner point clouds from FARO lidar scanners. Each scan includes an RGBD point-cloud and transformation data for registration. Check the has_laser_scanner_point_clouds column in the metadata CSV to verify availability for a specific video ID.

  6. Download ARKitScenes datasets using download_data.py

    main

    Use the download_data.py script to download specific datasets (3dod, upsampling, or raw). You can download data by specifying individual video_ids or by providing a CSV file containing split information.

    Download by Video ID

    To download specific videos, provide the dataset name, the --split (either Training or Validation), the --video_id (or multiple IDs), and the --download_dir.

    python3 download_data.py [3dod/upsampling/raw] --split [Training/Validation] --video_id video_id1 video_id2 --download_dir YOUR_DATA_FOLDER

    Download via CSV

    To download based on a CSV file, provide the dataset name and the --video_id_csv path. The script will automatically determine the splits from the CSV.

    python3 download_data.py [3dod/upsampling/raw] --video_id_csv CSV_PATH --download_dir YOUR_DATA_FOLDER

    Special Options for Raw Dataset

    • Laser Scanner Point Clouds: For the raw dataset, add the --download_laser_scanner_point_cloud flag.
    • Specific Assets: For the raw dataset, use --raw_dataset_assets to specify which types of data to download. Available assets include: mov, annotation, mesh, confidence, highres_depth, lowres_depth, lowres_wide.traj, lowres_wide, lowres_wide_intrinsics, ultrawide, ultrawide_intrinsics, vga_wide, and vga_wide_intrinsics.
    # Example: Download a specific video from the raw dataset
python3 download_data.py raw --split Training --video_id 47333462 --download_dir /tmp/ARKitScenes/

# Example: Download all assets specified in a CSV for the 3DOD dataset
python3 download_data.py 3dod --video_id_csv threedod/3dod_train_val_splits.csv --download_dir /tmp/raw_ARKitScenes/

# Example: Download specific assets from the raw dataset using a CSV
python3 download_data.py raw --video_id_csv raw/raw_train_val_splits.csv --download_dir /tmp/ar_raw_all/ --raw_dataset_assets mov annotation mesh confidence highres_depth lowres_depth lowres_wide.traj lowres_wide lowres_wide_intrinsics ultrawide ultrawide_intrinsics vga_wide vga_wide_intrinsics

  7. Setup the 3DOD virtual environment

    main

    To install the necessary packages for 3DOD, use the provided shell script to create a virtual environment in a venv/ folder within the threedod directory, then activate it.

    cd threedod
bash python_venv_setup.sh

source venv/bin/activate

  8. Train depth upsampling networks

    main

    Train upsampling networks using the train.py script. The script outputs metrics to the console every 5,000 iterations.

    Required Arguments:

    • --network: Choose between MSG or MSPF.
    • --upsample_factor: Choose an upsampling factor of 2, 4, or 8.

    Optional Arguments:

    • --tbp [port]: Specifies a TensorBoard port. Adding this will automatically launch a TensorBoard process in a subprocess to view training results.

    Example Training Command:

    python train.py --network MSG --upsample_factor 4
    python train.py --network [MSG/MSPF] --upsample_factor [2/4/8]

  10. Prepare single-frame (online) data

    main

    Use data_prepare_online.py to prepare datasets consisting of individual frames. This outputs separate point cloud and label files for each frame index.

    cd ./threedod/benchmark_scripts
python data_prepare_online.py \
       --data_root /tmp/raw_ARKitScenes/3dod/Training/ \
       --scene_id 41254925 \
       --gt_path /tmp/raw_ARKitScenes/3dod/Training/41254925/41254925_3dod_annotation.json \
       --output_dir /tmp/raw_ARKitScenes/3dod/Training/41254925/41254925_online_prepared_data/ \
       --vis

  12. Prepare whole-scene (offline) data

    main

    Use data_prepare_offline.py to accumulate point clouds from a video into a consistent 'world coordinate' system using camera intrinsics and extrinsics. This produces a single point cloud and a single label file for the entire scene.

    Output Format:

    • Point cloud: A N x 3 numpy array (.npy).
    • Labels: A dictionary (.npy) containing:
      • bboxes: (m, 7) array with (x, y, z, dx, dy, dz, theta).
      • types: List of category names.
      • uids: List of box IDs (optional).
      • pose: List of poses in each frame (optional).

    Note: To convert the 7-digit oriented bounding box codes to 8 corners, use boxes_to_corners_3d() in ./threedod/threedod_scripts/utils/box_utils.py.

    cd ./threedod/benchmark_scripts
python data_prepare_offline.py \
       --data_root /tmp/raw_ARKitScenes/3dod/Training/ \
       --scene_id 41254925 \
       --gt_path /tmp/raw_ARKitScenes/3dod/Training/41254925/41254925_3dod_annotation.json \
       --output_dir /tmp/raw_ARKitScenes/3dod/Training/41254925/41254925_offline_prepared_data/ \
       --vis