DepthCrafter

Follow these steps to set up the environment using uv:

1. Clone the repository:

   git clone https://github.com/Tencent/DepthCrafter.git

2. Navigate to the directory and set up the virtual environment:

   cd DepthCrafter
   uv venv
   source .venv/bin/activate
   uv sync

3. Verify the installation:

   uv pip list

DepthCrafter includes a suite of unit tests located in unit_tests/. You can run them using pytest.

Common Test Commands

• Run all tests: pytest unit_tests/
• Run with verbose output: pytest unit_tests/ -v
• Run a specific test file: pytest unit_tests/test_depth_crafter_ppl.py

Test Coverage

• test_depth_crafter_ppl.py: Main depth estimation pipeline.
• test_inference.py: Inference interface.
• test_utils.py: Utility functions.
• test_unet.py: UNet model.

Note: A GPU with CUDA support is required for test_pipeline_gpu_integration. Tests use small tensor sizes to minimize memory usage and mock heavy computations for speed.

Evaluation scripts are located in the benchmark folder.

1. Prepare Dataset: Run dataset_extract/dataset_extract_${dataset_name}.py to generate the necessary CSV files containing paths to RGB videos and depth .npz files.
2. Run Inference: Execute bash benchmark/infer/infer.sh. Note: You must replace input_rgb_root and saved_root with your actual paths in the script.
3. Run Evaluation: Execute bash benchmark/eval/eval.sh. Note: You must replace pred_disp_root and gt_disp_root with your actual paths in the script.

You can run depth estimation on videos using the run.py script. Depending on your GPU memory, choose between high-resolution or low-resolution modes.

High-resolution Inference

Requirements: ~26GB GPU memory (e.g., for 1024x576 resolution). Performance: ~2.1 fps on A100.

Low-resolution Inference

Requirements: ~9GB GPU memory (e.g., for 512x256 resolution). Performance: ~8.6 fps on A100.

Use the --max-res flag to specify the resolution for low-memory setups.

To launch a web-based user interface for DepthCrafter on your local machine, run the following command:

gradio app.py

Use the DepthCrafterInference class to set up the depth estimation pipeline. You must provide paths to the UNet model and the pre-trained pipeline. The class supports CPU offloading strategies to manage memory usage on GPUs.

Arguments:

• unet_path (str): Path to the UNet model.
• pre_train_path (str): Path to the pre-trained model.
• cpu_offload (Optional[str]): Strategy for CPU offloading. Options are:
  • "model": Enables model CPU offload.
  • "sequential": Enables sequential CPU offload.
  • None: No offloading (runs entirely on the specified device).
• device (str): Device to run the model on (e.g., `

The DepthCrafterPipeline is a specialized pipeline for generating consistent long depth sequences from input videos. It extends the StableVideoDiffusionPipeline and uses a sliding window approach with overlap to maintain temporal consistency across long video sequences.

To use the pipeline, call it with an input video (as a NumPy array or PyTorch tensor) and specify desired output dimensions and denoising parameters.

Use save_video to encode and save video frames to a file. It accepts a numpy array, a list of numpy arrays, or a list of PIL.Image.Image objects.

Arguments:

• video_frames: The frames to save. If providing a numpy array, it is expected to be in uint8 format; if it is float32, it will be automatically scaled by 255 and converted to uint8.
• output_video_path (str, optional): Path to save the video. If None, a temporary .mp4 file is created and its path is returned.
• fps (int): Frames per second. Defaults to 10.
• crf (int): Constant Rate Factor for encoding quality. Higher values mean lower quality/smaller size. Defaults to 18.

Returns:

• The path to the saved video file as a str.