VR-NeRF Eyeful Tower Dataset

The EyefulTower dataset consists of multiple scenes, each containing a specific number of cameras, positions, and images. The dataset is organized into different versions (v1, v2, v3) and provides data in various formats including 2K/1K EXRs and multiple resolutions of JPEGs (8K+, 4K, 2K, 1K).

Available scenes include:

• Residential/Living: apartment, kitchen, lounge_room, playroom (and variations: playroom_med, playroom_small), raf_furnishedroom.
• Office/Work: office1a, office1b, office1c, office2, office_view1, office_view2, meeting_room, workshop, workshop2.
• Specialized/Other: riverview, seating_area, table, vr_room, eyeful_room, raf_emptyroom.

Each scene has an associated index page (e.g., apartment_index) that provides a navigable interface for the data.

The VR-NeRF Eyeful Tower Dataset is a research dataset presented at ACM SIGGRAPH Asia 2023. It is associated with the VR-NeRF project and is designed for research involving neural radiance fields (NeRF) in virtual reality contexts.

Key resources for this project include:

• Project Page: https://vr-nerf.github.io
• Research Paper (arXiv): https://arxiv.org/abs/2311.02542
• Video Demonstration: https://youtu.be/9Q8_FssFQP4
• DOI: https://doi.org/10.1145/3610548.3618139

Commandly is a Godot-based control application designed for the remote control of the Eyeful Tower robot. It provides a comprehensive interface for teleoperation, real-time monitoring, and system management.

Key Capabilities:

• Dual Communication: Combines ROS# for ROS2 topic communication and SSH.NET for direct PAL system control.
• Joystick Control: Full gamepad support for robot teleoperation.
• Real-time Monitoring: Live camera feeds, IMU data, robot pose, and system status.
• Mapping & Localization: Tools to control and monitor SLAM and localization modules.
• Capture System: Integrated camera capture with configurable settings.
• Status Dashboard: Visual indicators for robot state, connectivity, and autonomous modes.

The repository provides 3D CAD models of the EyefulTower robot capture rig in various formats for different use cases, ranging from detailed sensor layouts to simplified geometry.

Available files include:

• EyefulRobot.STEP: Detailed 3D models of all cameras and sensors.
• EyefulRobot.zip: Native SolidWorks source files (.SLDPRT and .SLDASM) for the full robot assembly.
• Eyefulv2-elementary_geometry.STEP: A simplified 3D model of the robot, useful for lightweight simulations or collision checking.

The Eyeful repository is organized into the following components:

• app/: Platform-specific executables including macOS apps (mac/), Raspberry Pi apps (pi/), an SD card formatting utility (sd_format/), and a file transfer tool (transfer/).
• eyeful/: The core library for camera interaction (control, bracketing, multi-camera operations) using the Sony SDK.
• logging/: The logging framework.
• third-party/: External dependencies including cli11 (CLI parsing), fmt (text formatting), and magic_enum (enum reflection).

The CAD models are provided in two primary formats:

• .STEP: An open, vendor-neutral format (Standard for the Exchange of Product Data). Use this for compatibility with most CAD software such as SolidWorks, Fusion 360, or FreeCAD.
• .SLDPRT / .SLDASM: Native SolidWorks Part and Assembly files. These are the editable source files and require SolidWorks to open.

A typical capture sequence follows these steps:

1. Setup: Configure cameras for batch capture.
2. Drive Forward: Optionally move forward ~1.25 feet.
3. Wait for IMU: Wait for /eyeful/imu_ready to ensure stability.
4. Capture First Exposure: Trigger cameras with the first set of settings.
5. Capture Second Exposure: Apply second settings and capture again.
6. Spin: Rotate 120° (configurable via numSpinStops).
7. Repeat: Steps 3-6 repeat for each spin stop (default is 3 positions).
8. Record Position: Store the capture location for visualization.

The splits.json file defines which images are used for training and testing. To ensure valid evaluation, all images from a specific camera are held out for testing. The held-out cameras are:

• Eyeful v1: Camera 5
• Eyeful v2: Camera 17
• Eyeful v3: Camera 45

Each scene in the Eyeful Tower dataset follows a consistent directory structure. Key components include:

• cameras.json: Camera poses in KRT format.
• cameras.xml: Camera poses exported from Metashape.
• colmap/: COLMAP reconstruction files, including undistorted images in various resolutions (images, images_2, images_4, images_8) and the sparse reconstruction.
• images-1k/ & images-2k/: HDR images stored as uncompressed 32-bit floating-point EXR files, organized by camera ID.
• images-jpeg*: White-balanced and tone-mapped JPEG images at multiple resolutions (Full, 4K, 2K, 1K).
• mesh.*: Reconstructed 3D mesh in OBJ format with texture.
• splits.json: Defines the training and testing image sets.

The dataset contains images captured using three different versions of the Eyeful Tower rig:

• v1: 9 fisheye cameras.
• v2: 22 pinhole cameras (e.g., the 'kitchen' scene uses 19 cameras).
• v3: 14 fisheye cameras.

When selecting camera views for experimentation, it is recommended to view the collage video for a scene first to identify specific camera IDs (e.g., in the v2 apartment scene, camera IDs 19, 20, and 21 are at the same height).