Stable Virtual Camera (SEVA) Documentation

The benchmark provides 17 entries designed to evaluate Novel View Synthesis (NVS) models and facilitate fair comparison between different methods. The dataset includes various splits from different academic datasets (e.g., OmniObject3D, GSO, RealEstate10K, LLFF, DTU, CO3D, WildRGB-D, Mip-NeRF360, DL3DV-140, Tanks and Temples).

For each entry, the benchmark specifies:

• Split: The specific method/split used (e.g., S for SV3D, O for Ours, R for ReconFusion).
• Path: The directory location within the benchmark.
• Content: The file structure (e.g., train_test_split_*.json or images/*.png with transforms.json).
• Image Preprocessing: Operations applied to the model input (e.g., center crop to 576).
• Image Postprocessing: Operations applied to the model output before metric computation (e.g., resize to 256).

The model weights are hosted on Hugging Face and require authentication. You must first run the Hugging Face CLI login command, then request access on the official model card. Once access is granted, the code will handle the download automatically during the first run.

# This will prompt you to enter your Hugging Face credentials.
huggingface-cli login

To set up a development environment, install the package with the [dev] extra and install the pre-commit hooks.

# [OPTIONAL] Install seva dependencies for development.
pip install -e ".[dev]"
pre-commit install

For datasets that do not natively include images/*.png and transforms.json, you must convert them into the reconfusion format. This format is compatible with the ReconfusionParser in seva/data_io.py.

Critical Requirement: During conversion, you must sort the images using sorted(image_paths). This ensures that the images are correctly indexed by the provided train/test IDs.

An example script for this conversion process is provided in benchmark/export_reconfusion_example.py.

The Basic mode is the default interface designed for most users. It allows you to upload a single image and select from 13 preset camera trajectories to generate a video.

Preset Trajectories

• 3D Visualization: orbit, spiral, and lemniscate are effective for demonstrating the 3D structure of a scene.
• Focal Length Changes: zoom-in and zoom-out change the focal length while keeping the camera position static.
• Dolly Zoom: dolly zoom-in and dolly zoom-out change both the camera position (moving backward/forward) and the focal length.
• Directional Movement: move-forward, move-backward, move-up, move-down, move-left, and move-right change the camera's spatial position.

Note: move-forward and move-backward are omitted from the preset list in the UI for visualization purposes.

The CLI demo allows fine-grained control over the model, suitable for power users and researchers. The basic command structure is:

python demo.py --data_path <data_path> [additional arguments]

Key Configuration Concepts

• Procedural Two-Pass Sampling: It is recommended to enable this by setting --use_traj_prior True --chunk_strategy <chunk_strategy>. The choice of <chunk_strategy> depends on your specific task.
• Resolution and Aspect-Ratio: By default, images are center-cropped to $576\times 576$.
  • To set specific dimensions, use --W <W> --H <H>.
  • To maintain the original aspect ratio while resizing the shortest side to 576, use --L_short 576.

If you intend to use the CLI demo (demo.py) or the Gradio demo (demo_gr.py), you must initialize submodules and install additional dependencies for pycolmap and dust3r. Note that the core model does not depend on these, but they are required for the demo environments.

# Initialize and update submodules for demo.
git submodule update --init --recursive

# Install pycolmap dependencies for cli and gradio demo
echo "Installing pycolmap (for both cli and gradio demo)..."
pip install git+https://github.com/jensenz-sai/pycolmap@543266bc316df2fe407b3a33d454b310b1641042

# Install dust3r dependencies (only for gradio demo)
echo "Installing dust3r dependencies (only for gradio demo)..."
pushd third_party/dust3r
pip install -r requirements.txt
popd

To install the core seva model dependencies, run the following command from the repository root. This installs the package in editable mode.

pip install -e .

The Advanced mode provides fine-grained control over camera trajectories, making it suitable for power users and researchers.

In this mode, you can:

1. Upload one or multiple images.
2. Interact with a 3D viewport (powered by viser) to manually define the target camera trajectory.

To install the stable-virtual-camera repository, clone it recursively and install it in editable mode using pip.

Requirements:

• python >= 3.10
• torch >= 2.6.0
• Windows Users: Use WSL, as Flash Attention is not yet supported on native Windows.

For additional dependencies required for demos or development, refer to docs/INSTALL.md in the repository.

git clone --recursive https://github.com/Stability-AI/stable-virtual-camera
cd stable-virtual-camera
pip install -e .

You can potentially speed up sampling by up to 2x using torch.compile by installing the torch nightly version.

Warning: You must adjust the command to match your specific CUDA version. The example below is for CUDA 11.8.

# [OPTIONAL] Install the torch nightly version for faster JIT via. torch.compile
# Example for CUDA 11.8:
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu118