D2NeRF

D2NeRF is fully compatible with the HyperNeRF dataset format. To use your own video:

1. Process your video into a dataset using the Nerfies Capture Processing notebook.
2. Ensure the resulting data follows the Nerfies format used by HyperNeRF.

To plot telemetry to Tensorboard and render checkpoints on the fly during or after training, run the eval.py script using the same arguments as the training command.

Example Command:

python eval.py \
    --base_folder $EXPERIMENT_PATH \
    --gin_bindings="data_dir='$DATASET_PATH'" \
    --gin_configs $CONFIG_PATH

To start training, you need a dataset and an experiment path. Use the train.py script with Gin configuration bindings to specify the data directory.

Required Environment Variables/Paths:

• DATASET_PATH: Path to your unzipped dataset.
• EXPERIMENT_PATH: Path where you want to save the training results.
• CONFIG_PATH: Path to a .gin configuration file.

Example Command:

export DATASET_PATH=/path/to/dataset
export EXPERIMENT_PATH=/path/to/save/experiment/to
export CONFIG_PATH=configs/rl/001.gin
python train.py \
    --base_folder $EXPERIMENT_PATH \
    --gin_bindings="data_dir='$DATASET_PATH'" \
    --gin_configs $CONFIG_PATH

D2NeRF requires Python 3.8 or higher. Follow these steps to set up your environment:

1. Create a Conda environment:

   conda create --name d2nerf python=3.8

2. Install dependencies:

   pip install -r requirements.txt

3. Install the appropriate JAX distribution for your hardware (e.g., CUDA support):

   pip install --upgrade "jax[cuda]" -f https://storage.googleapis.com/jax-releases/jax_releases.html

D2NeRF uses Gin for configuration. The project provides several preset configuration directories:

• configs/decompose/: Template configurations defining shared configurations for NeRF and HyperNeRF.
• configs/rl/: Configurations for experiments on real-life scenes.
• configs/synthetic/: Configurations for experiments on synthetic scenes.

For detailed documentation on specific configuration parameters, refer to config.py in the repository.

The scene_gt.json file defines the spatial bounds and scaling of the scene. It requires:

• scale: A scaling factor derived from the 3D bounding box of the objects (e.g., 1.0 / sqrt(sum(bbox_diff**2))).
• center: The 3D center of the scene (often [0, 0, 0]).
• bbox: The 3D bounding box corners [[min_x, min_y, min_z], [max_x, max_y, max_z]].
• near and far: The near and far clipping planes for ray sampling.

# Example scene info generation
bbox_corners = np.array([[0, 0, 0], [50, 50, 10]])
scene_scale = 1.0 / np.sqrt(np.sum((bbox_corners[1] - bbox_corners[0]) ** 2))
scene_center = np.array([0., 0., 0.])

with scene_json_path.open('w') as f:
    json.dump({
        'scale': scene_scale,
        'center': scene_center.tolist(),
        'bbox': bbox_corners.tolist(),
        'near': 0.1,
        'far': 1.0,
    }, f, indent=2)

The parser generates two essential JSON files for dataset management:

1. dataset.json: Contains the total count of images, ids (filenames without extensions), and splits for train_ids and val_ids.
2. metadata.json: Maps each image_id to its corresponding warp_id, appearance_id, and camera_id. For standard datasets, these are often set to the index of the image in the training set.

For freeze tests, dataset.json is configured such that all images are in the val_ids set, and metadata.json maps each image_id to a specific freeze_frame ID.