DOCSEARCHbyAlgolia
Submit Docs

Depth Anything 3 (DA3)

repository·main·Indexed 26 days ago

https://github.com/bytedance-seed/depth-anything-3

A foundation model series for recovering spatially consistent geometry from monocular, multi-view, and pose-conditioned visual inputs. It supports depth estimation, 3D Gaussian estimation, and camera pose estimation. The library includes a Python API, CLI, and DA3-Streaming for processing image sequences, with various model families including Nested, Any-view, Monocular Metric, and Monocular Depth models.

Tokens
16.6K
Snippets
50
Records
84
Agent score
91%

What's inside Depth Anything 3

  1. Select a Depth Anything 3 Model

    main

    Depth Anything 3 offers several model families depending on your requirements for pose estimation, metric depth, and performance:

    • Nested Series: Uses an Any-view model for pose and depth estimation combined with a monocular metric depth estimator for scaling. Best for complex scenes.
      • Note: Prefer models with the -1.1 suffix (e.g., DA3NESTED-GIANT-LARGE-1.1) as they are retrained to fix previous bugs and offer better performance for street scenes.
    • Any-view Models: Models like DA3-GIANT and DA3-LARGE that provide relative depth and pose estimation.
    • Monocular Metric Depth: Specifically DA3METRIC-LARGE, which provides metric depth in meters and sky segmentation.
    • Monocular Depth: DA3MONO-LARGE for standard monocular depth estimation with sky segmentation.

    Model Summary Table:

    Model NameParamsRel. DepthPose Est.Pose Cond.GSMet. DepthSky Seg.License
    DA3NESTED-GIANT-LARGE-1.11.40B✅✅✅✅✅✅CC BY-NC 4.0
    DA3-GIANT-1.11.15B✅✅✅✅CC BY-NC 4.0
    DA3-LARGE-1.10.35B✅✅✅CC BY-NC 4.0
    DA3-BASE0.12B✅✅✅Apache 2.0
    DA3-SMALL0.08B✅✅✅Apache 2.0
    DA3METRIC-LARGE0.35B✅✅✅Apache 2.0
    DA3MONO-LARGE0.35B✅✅Apache 2.0

  2. Perform Basic Depth Estimation

    main

    To perform basic depth estimation, initialize the DepthAnything3 model using from_pretrained and pass a list of image paths to the inference() method.

    from depth_anything_3.api import DepthAnything3

# Initialize and run inference
model = DepthAnything3.from_pretrained("depth-anything/DA3NESTED-GIANT-LARGE").to("cuda")
prediction = model.inference(["image1.jpg", "image2.jpg"])

  3. Start the Depth Anything 3 backend service

    main

    The backend service allows you to keep the model resident in GPU memory, avoiding the overhead of reloading weights for every command. This is optional but recommended for repeated tasks.

    To start the service, use the da3 backend command and specify the model directory using --model-dir.

    da3 backend --model-dir depth-anything/DA3NESTED-GIANT-LARGE

  4. Process Images, Videos, and COLMAP Datasets

    main

    Use specific subcommands for direct processing without auto-detection:

    Process a single image

    da3 image image.jpg --export-dir ./output1 --use-backend

    Process a video

    da3 video video.mp4 --fps 2.0 --export-dir ./output2 --use-backend

    Process a COLMAP dataset

    da3 colmap ./colmap_data --export-dir ./output3 --use-backend
    da3 image image.jpg --export-dir ./output1 --use-backend

da3 video video.mp4 --fps 2.0 --export-dir ./output2 --use-backend

da3 colmap ./colmap_data --export-dir ./output3 --use-backend

  5. Download and extract Visual Geometry Benchmark datasets

    main

    To run the benchmark, you must first download the datasets from HuggingFace. It is recommended to install the HuggingFace CLI first: pip install -U huggingface_hub[cli]. If downloads are slow, you can use the mirror by setting export HF_ENDPOINT=https://hf-mirror.com.

    Follow these steps to download and extract all datasets into a workspace directory:

    cd da3_release

# Create directory and download from HuggingFace
mkdir -p workspace/benchmark_dataset
hf download depth-anything/DA3-BENCH \
    --local-dir workspace/benchmark_dataset \
    --repo-type dataset

# Extract all datasets
cd workspace/benchmark_dataset
for f in *.zip; do unzip -q "$f"; done

  6. Run the Depth Anything 3 evaluation pipeline

    main

    The evaluation pipeline can be run using the depth_anything_3.bench.evaluator module. You can perform full evaluations, skip inference to evaluate existing predictions, or simply print results.

    Note: The default model is depth-anything/DA3-GIANT.

    # Set model
MODEL=depth-anything/DA3-GIANT

# Full evaluation (inference + evaluation + print results)
python -m depth_anything_3.bench.evaluator model.path=$MODEL

# Skip inference, only evaluate existing predictions
python -m depth_anything_3.bench.evaluator eval.eval_only=true

# Only print saved metrics
python -m depth_anything_3.bench.evaluator eval.print_only=true

  7. Reuse a running backend for processing jobs

    main

    To submit a job to an already running da3 backend service via HTTP, use the --use-backend flag and provide the service address with --backend-url.

    Important Note on --export-dir: When using --use-backend, the path provided to --export-dir must be a relative path located under the backend's --gallery-dir (which defaults to workspace/gallery). For example, if your backend uses workspace/gallery, you should use --export-dir workspace/gallery/scene002.

    da3 auto path/to/video.mp4 \
    --export-dir workspace/gallery/scene002 \
    --use-backend \
    --backend-url http://localhost:8008

  8. Calculate Metric Depth from DA3METRIC-LARGE

    main

    To convert the output of DA3METRIC-LARGE into metric depth in meters, use the following formula:

    metric_depth = focal * net_output / 300.

    Where:

    • focal is the focal length in pixels (typically the average of fx and fy from the camera intrinsic matrix K).

    Note: The output from DA3NESTED-GIANT-LARGE is already provided in meters and does not require this conversion.

    metric_depth = focal * net_output / 300.

  9. Use Auto Mode to Process Inputs

    main

    The auto command automatically detects the input type (image, video, or dataset) and processes it.

    Basic Auto Mode

    da3 auto ./path/to/input --export-dir ./output

    Auto Mode with Backend Acceleration

    To speed up processing, run a backend service first, then use the --use-backend and --backend-url flags:

    da3 auto ./path/to/input \
    --use-backend \
    --backend-url http://localhost:8008 \
    --export-dir ./output
    da3 auto ./unknown_input --export-dir ./output

da3 auto ./unknown_input \
    --use-backend \
    --backend-url http://localhost:8008 \
    --export-dir ./output

  10. Configure Reference View Selection Strategy

    main

    When performing multi-view depth estimation with 3 or more views, you can specify a ref_view_strategy to determine which view serves as the primary reference frame. This choice affects the quality and consistency of depth predictions.

    Available Strategies

    StrategyDescriptionRecommended Use Case
    saddle_balanced(Default) Selects a view that balances similarity, feature norm, and feature variance.General purpose, unknown scenarios, or diverse photo sets.
    saddle_sim_rangeSelects the view with the largest similarity range (max - min) to other views.Wide-baseline multi-view setups to maximize information coverage.
    middleSelects the view at the middle index of the input sequence.Video sequences or temporally ordered frames (e.g., DA3-LONG) where the middle frame has maximum overlap.
    firstAlways selects the first view (index 0).Not recommended unless you have manually pre-sorted views to ensure the first is optimal.

    Note: For 1 or 2 views, no reordering is performed (effectively using the first view).

    ### Python API Example

```python
from depth_anything_3 import DepthAnything3

model = DepthAnything3.from_pretrained("depth-anything/DA3NESTED-GIANT-LARGE")

# Use default (saddle_balanced)
prediction = model.inference(
    images,
    ref_view_strategy="saddle_balanced"
)

# For video sequences
prediction = model.inference(
    video_frames,
    ref_view_strategy="middle"
)

# For wide-baseline multi-view
prediction = model.inference(
    images,
    ref_view_strategy="saddle_sim_range"
)

    CLI Example

    # Default (saddle_balanced)
da3 auto input/ --export-dir output/

# Explicitly specify strategy
da3 auto input/ --ref-view-strategy saddle_balanced

# For video processing
da3 video input.mp4 --ref-view-strategy middle

# For wide-baseline multi-view
da3 images captures/ --ref-view-strategy saddle_sim_range

  11. Run evaluation with Multi-GPU inference

    main

    The evaluator automatically distributes inference across available GPUs. You can control GPU usage via the CUDA_VISIBLE_DEVICES environment variable.

    # Use 4 specific GPUs
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m depth_anything_3.bench.evaluator model.path=$MODEL

# Use all available GPUs (default)
python -m depth_anything_3.bench.evaluator model.path=$MODEL

# Single GPU
CUDA_VISIBLE_DEVICES=0 python -m depth_anything_3.bench.evaluator model.path=$MODEL

  12. Install DA3-Streaming

    main

    To set up DA3-Streaming, follow these steps to clone the repository, install dependencies, and download weights.

    1. Clone the repository

    Use the --recursive flag to ensure submodules are included:

    git clone --recursive https://github.com/ByteDance-Seed/Depth-Anything-3.git

    If you have already cloned without submodules, run:

    cd <your_dir>/Depth-Anything-3/
git submodule update --init --recursive .

    2. Environment Setup

    Install Python dependencies

    First, install the requirements for Depth-Anything-3:

    pip install -r requirements.txt

    Download pre-trained weights

    Run the provided script to download necessary weights:

    bash ./scripts/download_weights.sh

    System dependencies

    If you encounter errors regarding libGL.so.1 (common with opencv-python), install the following system package:

    sudo apt-get install -y libgl1-mesa-glx
    git clone --recursive https://github.com/ByteDance-Seed/Depth-Anything-3.git