DyCheck: Framework for Evaluating Monocular Dynamic View Synthesis

The DYCHECK dataset extends the standard Nerfies format with additional files for co-visibility, depth, keypoints, and metadata.

Directory Structure:

<DYCHECK_ROOT>/datasets/<DATASET>/<SEQUENCE>/
    ├── camera/
    │   └── <ITEM_ID>.json
    ├── **covisible/**
    │   └── <SCALE>x
    │       └── <SPLIT>
    │           └── <ITEM_ID>.png
    ├── **depth/**
    │   └── <SCALE>x
    │       └── <ITEM_ID>.npy
    ├── **keypoint/**
    │   └── <SCALE>x
    │       └── <SPLIT>
    │           ├── <ITEM_ID>.json
    │           └── skeleton.json
    ├── rgb/
    │   └── <SCALE>x
    │       └── <ITEM_ID>.png
    ├── **splits/**
    │   └── <SPLIT>.json
    ├── dataset.json
    ├── **emf**.json
    ├── **extra**.json
    ├── metadata.json
    ├── points.npy
    └── scene.json

Note: Items marked with ** are additional data provided by DYCHECK.

To set up the combined Nerfies-HyperNeRF dataset (7 sequences total) and inject DYCHECK's additional data, use the provided shell script. This script handles the removal of the vrig- prefix from HyperNeRF sequences to focus on validation camera rigs.

# Download Nerfies and HyperNeRF datasets and inject our additional data.
bash scripts/download_nerfies_hypernerf_datasets.sh <DRIVE_NAME>

Follow these steps to install the required system libraries and Python packages.

System Dependencies: Install libopenexr-dev and ffmpeg via apt.

Python Dependencies: Install the requirements via pip. This installs the repository as an editable local package, meaning changes to the source code will propagate to your environment automatically.

CUDA Compatibility Note: The default requirements.txt is tailored for CUDA 11.5 and 11.7. If using different CUDA versions, manually update jax, jaxlib, torch, and torchvision in the requirements file following their respective installation guides.

# System level `tensorflow-graphics` dependency.
sudo apt install libopenexr-dev ffmpeg -y

# Main project dependencies.
pip install -r requirements.txt

If you intend to use the project's annotation tools, you must install specific JupyterLab extensions and register the project's Python environment as a Jupyter kernel.

# Setup labextensions for annotation and visualization.
jupyter labextension install jupyterlab-plotly ipyevents

# Register the project environment as a kernel.
python -m ipykernel install --user --name=dycheck

To use the provided scripts for downloading datasets and checkpoints from Google Drive, you must configure rclone.

1. Run rclone config to set up your Google Drive profile.
2. Verify access to the dycheck release folder using the lsd command.

# Set up `rclone` with your google drive.
rclone config

# Verify access (replace <DRIVE_NAME> with your rclone profile name)
rclone lsd --drive-shared-with-me "<DRIVE_NAME>:/dycheck-release"

To quickly test the repository, you can run a demo task using demo/launch.py. A minimal dataset and pre-trained T-NeRF model are provided in the demo folder. You can specify different tasks such as novel_view, stabilized_view, or bullet_time via the --task argument.

# Launch a demo task.
python demo/launch.py --task <TASK>

For multi-camera setups, you must synchronize the testing cameras with the training camera using audio-based temporal synchronization.

1. Combine AV channels: Use tools/combine_record3d_av.py to prepare the audio/video files.
2. Determine Offset: Use Adobe Premiere Pro to create a multi-camera source sequence. Identify the frame offset between each testing video and the training video.
3. Record Offset: Save the integer offset into a file named timesync_offset.txt within the testing camera's RGBD_Video directory.

python tools/combine_record3d_av.py \
   --gin_configs configs/iphone/combine_record3d_av.gin \
   --gin_bindings 'combine_record3d_av.Config.data_root="/shared/hangg/datasets/iphone/record3d"' \
   --gin_bindings 'SEQUENCE="<SEQUENCE>"'

# After finding the offset in Premiere Pro:
echo <OFFSET> > <TEST_VIDEO_ID>/RGBD_Video/timesync_offset.txt

Use the provided shell scripts to download checkpoints via rclone. You can download all available checkpoints at once or select a specific model, dataset, sequence, and setting.

Checkpoint Directory Structure: All checkpoints are organized as: <DYCHECK_ROOT>/work_dirs/<DATASET>/<SEQUENCE>/<MODEL>/<SETTING>

# Download all checkpoints
bash scripts/download_all_checkpoints.sh <DRIVE_NAME>

# Download a specific checkpoint
bash scripts/download_all_checkpoints.sh <DRIVE_NAME> <DATASET> <SEQUENCE> <MODEL> <SETTING>

Convert your raw Record3D capture into the iPhone dataset format using the provided shell script. You may need to edit scripts/process_record3d_to_iphone.sh to specify:

• Start and end frames (for trimming).
• Foreground prompts for segmentation.
• Rotation mode (for image orientation).
• Depth filtering parameters.

bash scripts/process_record3d_to_iphone.sh <SEQUENCE>