nerfview

To integrate nerfview into your project, you need to define a render_fn and initialize a nerfview.Viewer using a viser.ViserServer.

Your render_fn must accept two arguments:

1. camera_state (nerfview.CameraState): Provides the camera-to-world matrix (c2w) and an intrinsic matrix method (get_K).
2. render_tab_state (nerfview.RenderTabState): Provides information about the current render mode (e.g., preview_render) and the target resolution (render_width, render_height, viewer_width, viewer_height).

The render_fn must return a uint8 numpy array representing the image.

from typing import Tuple
import viser
import nerfview
import numpy as np

def render_fn(
    camera_state: nerfview.CameraState, render_tab_state: nerfview.RenderTabState
) -> np.ndarray:
    # Determine resolution based on whether it is a preview render
    if render_tab_state.preview_render:
        width = render_tab_state.render_width
        height = render_tab_state.render_height
    else:
        width = render_tab_state.viewer_width
        height = render_tab_state.viewer_height

    # Extract camera parameters as float64 numpy arrays
    c2w = camera_state.c2w
    K = camera_state.get_K([width, height])

    # Perform your rendering logic here
    img = your_rendering_logic(c2w, K, width, height) 
    
    # Return image as uint8 numpy array
    return img

# Initialize the server and viewer
server = viser.ViserServer(verbose=False)
viewer = nerfview.Viewer(server=server, render_fn=render_fn, mode='rendering')

The CameraPath class is used to manage a sequence of keyframes that define a camera trajectory. It uses Kochanek-Bartels splines to interpolate between keyframes for smooth movement, rotation (via quaternions), field-of-view (FOV), and optionally time.

Key capabilities include:

• Adding Keyframes: Use add_camera(keyframe) to insert a new pose. You can also replace an existing keyframe by providing its keyframe_index.
• Interpolation: Use interpolate_pose_and_fov_rad(normalized_t) to get the interpolated tf.SE3 pose and FOV at a given normalized time t (0.0 to 1.0).
• Trajectory Control: Supports looping trajectories and adjusting spline tension for smoothness.
• Visualization: Can visualize the camera path as a spline and show keyframe frustums in the viser scene.

from nerfview.render_panel import CameraPath, Keyframe
import numpy as np

# Assuming 'server' is an initialized viser.ViserServer
# and 'duration_element' is a viser.GuiInputHandle[float]
camera_path = CameraPath(server, duration_element)

# Create a keyframe manually
new_keyframe = Keyframe(
    position=np.array([0, 0, 0]),
    wxyz=np.array([1, 0, 0, 0]),
    override_fov_enabled=False,
    override_fov_rad=0.785,
    override_time_enabled=False,
    override_time_val=0.0,
    aspect=1.0,
    override_transition_enabled=False,
    override_transition_sec=None
)

camera_path.add_camera(new_keyframe)

# Interpolate at halfway point
result = camera_path.interpolate_pose_and_fov_rad(0.5)
if result:
    pose, fov = result[:2]
    print(f"Pose: {pose}, FOV: {fov}")

When mode="training" is used, the Viewer provides additional GUI controls for managing the training process and integrates with the training loop via the update method.

Key features in training mode:

• Pause/Resume: Controls the training state.
• Train Util Slider: Adjusts the balance between training rays and view rays.
• update(step, num_train_rays_per_step): This method should be called within your training loop. It calculates how often the viewer should request a new render based on the user's train_util setting and the requested ray counts to ensure smooth viewing without stalling training.

# Inside your training loop
for step, (images, rays) in enumerate(dataloader):
    # ... perform training step ...
    
    # Update the viewer
    # num_train_rays_per_step is the number of rays used in this specific step
    viewer.update(step, num_train_rays_per_step=len(rays))

The repository includes several examples for different use cases. You can run them using the following commands:

• Dummy Rendering (Basic API): python examples/00_dummy_rendering.py
• Dummy Training (Training-time updates): python examples/01_dummy_training.py
• Mesh Rendering (using nvdiffrast): Requires downloading assets first via bash examples/assets/download_dragon_mesh.sh.
• Pretrained 3DGS (using gsplat): Requires downloading assets first via bash examples/assets/download_gsplat_ckpt.sh.
• 3DGS Training (using gsplat): Requires downloading assets first via bash examples/assets/download_colmap_garden.sh.

# Example: Mesh Rendering
bash examples/assets/download_dragon_mesh.sh
CUDA_VISIBLE_DEVICES=0 python examples/02_mesh_rendering.py

# Example: Pretrained 3DGS
bash examples/assets/download_gsplat_ckpt.sh
CUDA_VISIBLE_DEVICES=0 python examples/03_gsplat_rendering.py --ckpt examples/assets/ckpt_6999_crop.pt

The Viewer class is the primary interface for interactive web viewing. When instantiated, it binds to a viser.ViserServer, creates a GUI based on the specified mode, and manages background rendering tasks.

To use it, you must provide a render_fn that handles the actual rendering logic. This function is called by the viewer whenever the camera moves or a re-render is requested.

import viser
import numpy as np
from pathlib import Path
from nerfview.viewer import Viewer

# 1. Define your rendering function
def my_render_fn(camera_state, img_wh):
    # camera_state is a CameraState object
    # img_wh is a tuple (W, H)
    # Return a uint8 numpy array (RGB) or (RGB, float32 depth)
    image = np.zeros((img_wh[1], img_wh[0], 3), dtype=np.uint8)
    return image

# 2. Setup viser server
server = viser.ViserServer()

# 3. Initialize Viewer
v = Viewer(
    server=server,
    render_fn=my_render_fn,
    output_dir=Path("./my_results"),
    mode="rendering"  # or "training"
)