Vista4D

To perform video reshooting inference using DSE, use the example_inference_dse.sh script. Outputs are saved to ./results/dse/$EXAMPLE/vista4d_$RESOLUTION/.

Multi-GPU Support: USP multi-GPU inference is supported by setting USE_USP=true and specifying the number of GPUs via NUM_GPUS.

EXAMPLE=lounge-cup RESOLUTION=720p bash scripts/inference/example_inference_dse.sh

After setting up the Conda environment and CUDA toolkit, install PyTorch (version 2.10.0 tested) specifically for CUDA 12.8, followed by the project's requirements.

pip3 install torch==2.10.0 torchvision==0.25.0 --index-url https://download.pytorch.org/whl/cu128
pip3 install -r requirements.txt

Run inference on rendered point clouds using the example_inference_single.sh script. Outputs are stored in ./results/single/$EXAMPLE/vista4d_$RESOLUTION/.

Unified Sequence Parallel (USP): To speed up inference and reduce per-GPU VRAM usage via multi-GPU inference, use the USE_USP=true flag. You can specify the number of GPUs with NUM_GPUS=? (defaults to 8).

Execution:

EXAMPLE=couple-newspaper RESOLUTION=720p bash scripts/inference/example_inference_single.sh

# Standard inference
EXAMPLE=couple-newspaper RESOLUTION=720p bash scripts/inference/example_inference_single.sh

# Multi-GPU inference using USP
USE_USP=true NUM_GPUS=8 EXAMPLE=couple-newspaper RESOLUTION=720p bash scripts/inference/example_inference_single.sh

The Camera UI supports DSE reconstructions. To load them, enter the reconstruction directory (e.g., ./results/dse/$EXAMPLE/recon_and_seg/) into the Folder path field and click Load.

While the editable timeline only covers source frames, two DSE-specific controls are available:

• Show DSE cameras: Overlays the scene-capture camera frustums and their paths.
• DSE frame interval: Strides through DSE frames when rebuilding the static background overlay. Increasing this value makes the overlay sparser.

To set up the Vista4D environment, create a new Conda environment with Python 3.12 and install the CUDA 12.8 toolkit. It is recommended to install a self-contained CUDA toolkit into the environment to ensure compatibility. You must also set CUDA_HOME and update LD_LIBRARY_PATH to point to the Conda environment's paths.

conda create --name vista4d python=3.12
conda activate vista4d

# Install CUDA 12.8 and C++ compiler
conda install -c nvidia cuda-toolkit=12.8
conda install -c conda-forge gxx_linux-64

# Set environment variables
export CUDA_HOME=$CONDA_PREFIX
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH

Vista4D inference requires both the Vista4D finetuned checkpoints and the base Wan2.1-T2V-14B model.

Vista4D Checkpoints: Download to ./checkpoints/vista4d/.

• To download all: hf download Eyeline-Labs/Vista4D --local-dir ./checkpoints/vista4d
• To download a specific resolution:
  • 384p: hf download Eyeline-Labs/Vista4D --local-dir ./checkpoints/vista4d --include "384p49_step=30000/*"
  • 720p: hf download Eyeline-Labs/Vista4D --local-dir ./checkpoints/vista4d --include "720p49_step=3000/*"

Wan 2.1 Base Model: Download to ./checkpoints/wan/Wan2.1-T2V-14B/.

• hf download Wan-AI/Wan2.1-T2V-14B --local-dir ./checkpoints/wan/Wan2.1-T2V-14B

# Download Vista4D (all variants)
hf download Eyeline-Labs/Vista4D --local-dir ./checkpoints/vista4d

# Download Wan 2.1 base model
hf download Wan-AI/Wan2.1-T2V-14B --local-dir ./checkpoints/wan/Wan2.1-T2V-14B

Run inference on the evaluation dataset using the provided script. The script uses metadata.csv for prompts and seeds.

Multi-GPU Sharding: To speed up inference across multiple GPUs, use NUM_SHARDS and SHARD_ID to split the workload. Each shard processes a contiguous chunk and automatically skips already-completed pairs.

Example for running the 3rd shard (index 2) out of 8 on a specific GPU:

NUM_SHARDS=8 SHARD_ID=2 RESOLUTION=720p bash scripts/inference/example_inference_eval.sh

To unproject the source clip and DSE frames into a single 4D point cloud and render them in target cameras, use the example_render_dse.sh script.

Outputs are saved to ./results/dse/$EXAMPLE/render_$RESOLUTION/.

Configuration Options:

• RESOLUTION: Set the target resolution (e.g., 720p).
• DSE_FRAME_INTERVAL: Controls subsampling of DSE frames during unprojection. The default is 4. A higher value means fewer DSE frames are used.
• RENDER_ONLY_NECESSARY: Can be configured similarly to single-video rendering.

EXAMPLE=lounge-cup RESOLUTION=720p bash scripts/preprocess/example_render_dse.sh

To unproject the 4D reconstruction into a point cloud and render it using target cameras, use the example_render_single.sh script.

Options:

• EXAMPLE: The source video name (e.g., couple-newspaper).
• RESOLUTION: Supports 384p or 720p (matches model checkpoints).
• RENDER_ONLY_NECESSARY: Defaults to true. Set to false to output additional renders (without temporal persistence or double-reprojected versions) for baselines/ablations.

Results are stored in ./results/single/$EXAMPLE/render_$RESOLUTION/.

# Standard render
EXAMPLE=couple-newspaper RESOLUTION=720p bash scripts/preprocess/example_render_single.sh

# Render with all outputs (for ablations/baselines)
EXAMPLE=couple-newspaper RESOLUTION=720p RENDER_ONLY_NECESSARY=false bash scripts/preprocess/example_render_single.sh

Dynamic Scene Expansion (DSE) allows Vista4D to incorporate extra visual information (like casual scene captures or alternate angles) into a single temporally-persistent 4D point cloud. This reduces hallucinations by providing more context to the video diffusion model.

To jointly reconstruct and segment a source clip and its paired scene capture, use the example_recon_and_seg_dse.sh script. Results, including a clips.json file that records source and DSE frame ranges, are saved to ./results/dse/$EXAMPLE/recon_and_seg/.

EXAMPLE=lounge-cup RECON_METHOD=pi3 bash scripts/preprocess/example_recon_and_seg_dse.sh