WildDet3D

The WildDet3D iPhone app provides real-time on-device 3D object detection using ARKit and LiDAR depth. Key features include:

• Camera mode: AR-based 3D detection with LiDAR depth alignment.
• Upload mode: Detect objects in existing photos from your library.
• RGB / RGBD modes: Supports both monocular (RGB) and depth-enhanced (RGBD) detection.
• Interactive boxes: Allows you to tap to select, view dimensions, or delete individual detections.
• Open-vocabulary: Enables detection of any object category via text prompts.

The evaluation reports several key metrics to assess 3D detection performance:

• AP (Average Precision): COCO-style 3D IoU. For ScanNet, Argoverse2, In-the-Wild, and DROID, the evaluator uses center-distance (dist / prox) matching.
• ATE / ASE / AOE: Mean translation, scale, and orientation error of matched detections.
• ODS / ODS_Sym / ODS_Canonical: nuScenes-style Open Detection Score. ODS_Canonical uses canonical-rotation AOE and is the primary ranking metric for open-vocabulary benchmarks.
• AP_<Subset>: Specific AP reported for Omni3D sub-datasets (e.g., KITTI, nuScenes, SUNRGBD, etc.).
• Base / Novel Split: Evaluators distinguish between 'Base' (canonical categories like the 15 SUNRGBD indoor categories or 11 AV2 driving categories) and 'Novel' (everything else).

To ensure smooth trajectories, the pipeline applies two different smoothing techniques:

1. Kalman Filter: Applied to the 3D center position and dimensions. The state vector is [cx, cy, cz, w, l, h, vx, vy, vz] (position + dimensions + velocity), and the observation is [cx, cy, cz, w, l, h] from WildDet3D.
2. Exponential Moving Average (EMA): Applied separately to rotation quaternions.

Post-processing steps:

• Yaw normalization: Quaternion yaw is normalized to [0, pi) to resolve 180-degree ambiguity.
• 90-degree flip fix: For near-square objects (where $w/l > 0.7$), the pipeline resolves 90-degree symmetry ambiguity to ensure temporal yaw consistency.

The integration relies on the wilddet3d_to_obb() adapter in run_wilddet3d.py to align WildDet3D outputs with Boxer's ObbTW requirements. Two key alignments are performed:

1. Pose Chain: WildDet3D returns 3D boxes in the camera frame. Boxer expects them in the world frame. The transformation used is: T_world_camera = T_world_rig @ T_camera_rig.inverse(), utilizing Boxer's cam.T_camera_rig attribute.

2. Axis Convention: WildDet3D's RoI2Det3D outputs [center(3), dims=(W,L,H)(3), quat_wxyz(4)] with a local box frame of x=L, y=H, z=W. Boxer's bb3_object keeps the vertical axis on z. To align them, the adapter rotates the box 90 degrees around z and maps dimensions as bb3 = (L=x, W=y, H=z). This allows the quaternion to be used as-is without extra rotation, ensuring boxes align correctly with vertical objects like walls or paintings.

The VLM-driven demo uses a Vision-Language Model to translate natural language queries (e.g., "Detect all the sheep in this image") into spatial prompts that guide WildDet3D. There are two supported modes based on the VLM used:

• box mode: Uses Qwen3-VL-8B to generate 2D bounding boxes via tool calling.
• point mode: Uses Molmo2-8B to generate 2D points via native pointing.

Files in the FoundationPose GSO dataset follow specific naming patterns based on whether they are per-view (camera-specific) or per-scene.

Per-view files: {group_id}_scene_{scene_id:08d}_cam_{cam_id}.{ext}

• group_id: Numeric scene group identifier (e.g., 1004491151)
• scene_id: 8-digit zero-padded scene index (e.g., 00000412)
• cam_id: Camera index (0 or 1)
• Example: 4280099961_scene_00000412_cam_1.png

Per-scene files: {group_id}_scene_{scene_id:08d}_states.json

WildDet3D uses a progressive 3-stage training pipeline to scale from indoor/outdoor benchmarks to diverse in-the-wild data:

1. Stage 1: Omni3D Canonical (12 epochs): Trains on Omni3D with canonical rotation using a 5-mode collator (text + box geometry prompts).
2. Stage 2: All-Data Dense Finetune (12 epochs): Uses the Stage 1 checkpoint to train on 8 datasets (human-only ITW + V3Det) using the 5-mode collator (text + box geometry prompts; no mask).
3. Stage 3: High-Quality Mix-Prompt Finetune (3 epochs): Uses the Stage 2 checkpoint to finetune on a high-quality mix (90% Omni3D + 10% ITW human) using the mask_pt collator (text + box + point prompts; points sampled from masks).

To link semantic IDs from masks to actual objects, use the files in annotations/instance_mappings/gso/.

• mapping.json: Maps semantic IDs (from mask pixels) to USD prim paths.
• semantics.json: Maps semantic IDs to lowercased class names.

Example Mapping:

• Mask pixel 10 $\rightarrow$ mapping.json $\rightarrow$ /World/objects/gso_Perricone_MD_Nutritive_Cleanser/model/mesh.

// mapping.json
{
  "0": "BACKGROUND",
  "10": "/World/objects/gso_Perricone_MD_Nutritive_Cleanser/model/mesh"
}

WildDet3D evaluation is performed using the vis4d framework. To run an evaluation, use the vis4d test command, specifying the benchmark configuration file and your model checkpoint.

General Command Format:

vis4d test --config configs/eval/<benchmark>/<mode>.py --gpus 1 --ckpt <path_to_checkpoint>

vis4d test --config configs/eval/<benchmark>/<mode>.py \
    --gpus 1 --ckpt ckpt/wilddet3d.pt