DOCSEARCHbyAlgolia
Submit Docs

realtime-calib Documentation

website·Indexed 29 days ago

https://realtime-calib.hans-brgs.dev

A local, real-time, open-source multi-camera calibration application for recovering intrinsic and extrinsic calibration. The tool is orchestrated with Docker and provides features for defining calibration boards, estimating focal length and distortion, and exporting 6-DoF pose data.

Tokens
8.1K
Snippets
8
Records
52
Agent score
55%

What's inside realtime-calib

  1. Overview of realtime-calib

    realtime-calib is a local, real-time, open-source application used to recover intrinsics (focal length, distortion) and extrinsics (6-DoF position and orientation) for USB camera rigs. It provides live feedback via a web app accessible to any device on the same local network as the server. It is designed to run headless (e.g., in Docker on Linux VMs) and supports CPU-only operation without requiring a GPU or cloud connection.

  2. Calibration Workflow Overview

    The calibration process follows a sequential wizard. Steps remain locked until the previous requirements are met:

    1. Session Management: Start a "New realtime calibration" or "Load from files" to unlock the wizard.
    2. Target Config: Define the calibration board (e.g., ChArUco). This must be done before Camera Setup.
    3. Camera Setup: Discover USB cameras and configure live previews.
    4. Intrinsic Calibration: Perform per-camera sweeps, record clips, trim/sample frames, and compute results.
    5. Extrinsic Calibration: Use synchronized frame groups across camera pairs to determine poses, refine with bundle adjustment, and review in 3D.
    6. Export: Export results as Caliscope-native TOML or engine-ready JSON (three.js, Blender, Unity, Unreal).

  3. Understand realtime-calib accuracy metrics and reproducibility

    The project tracks accuracy through several key metrics, including per-view and aggregate reprojection error for intrinsics, extrinsic consistency after bundle adjustment, and the runtime/latency of the real-time path. To ensure reproducibility, every published accuracy number is linked to the specific dataset, the exact commit/tag of the code, and the command used to generate the result.

  4. realtime-calib System Architecture Overview

    realtime-calib is a multi-camera calibration system where heavy processing (capture, detection, and calibration) occurs on a server connected to the cameras, while the operator controls the process via a responsive web app over a local network. The system is orchestrated with Docker Compose and uses a dual-channel communication strategy: transactional commands are handled via an HTTP API, and continuous media/telemetry (video tracks with server-side overlays and quality data) are handled via WebRTC using LiveKit.

  5. Select keyframes for real-time calibration

    To achieve single-pass, real-time behavior, realtime-calib selects a diverse subset of detections (default cap of 25) using the following pipeline:

    1. Stride: Decimates high-fps runs by keeping one detection every N frames.
    2. Quality Gates: Drops frames based on Laplacian-variance sharpness (blurriness), insufficient ChArUco corner counts, or degenerate/poorly spread corners.
    3. Farthest-point sampling: Candidates are described by a 3D feature consisting of board tilt (0–45°, normalized) and detection centroid (x, y as image fractions). The algorithm starts with the frame containing the most corners and greedily adds the candidate farthest from those already selected to maximize orientation and sensor-region coverage.

  6. Perform extrinsic camera calibration

    The 6-DoF pose for every camera is recovered in a single shared coordinate frame (the anchor camera, index 0, fixed as identity) through these steps:

    1. Pairwise Poses: Relative transforms for co-visible camera pairs are estimated using cv2.stereoCalibrate on shared board views in normalized coordinates.
    2. Chaining: Poses are chained from the anchor along the lowest-cumulative-error path using Dijkstra's algorithm.
    3. Triangulation: Board corners are triangulated into a 3D point cloud using Direct Linear Transformation (DLT) over all observing rays.
    4. Bundle Adjustment: scipy.optimize.least_squares (using the 'trf' method and a sparse Jacobian) jointly refines non-anchor poses and 3D points to minimize reprojection error.

  7. Perform one-time setup for realtime-calib

    Before launching the stack, perform the following setup steps to configure the environment and generate necessary TLS certificates for LAN access.
    # 1. Create environment file and set HOST_IP to the machine's LAN IP (e.g. 192.168.1.42)
cp .env.example .env

# 2. Trust the mkcert local CA (once per machine)
mkcert -install

# 3. Generate the LAN certificate (covers HOST_IP + localhost)
./caddy/generate-certs.sh

  8. Select and prepare a calibration board

    For optimal camera calibration, use a ChArUco board. This combines ArUco markers (which identify corners and prevent rotation ambiguity/occlusion issues) with chessboard corners for subpixel accuracy.

    Board Selection & Preparation:

    • Dictionary: Use the smallest ArUco dictionary that meets your marker needs to maximize Hamming distance and error correction, reducing false detections.
    • Geometry: Use larger squares for longer working distances and denser grids for wide-angle/fisheye lenses to capture edge data. A common default is a 7x8 grid with a 0.75 marker-to-square ratio.
    • Physicality: Mount the board on a rigid, flat substrate to avoid systematic errors from warped paper.
    • Scaling: Do not rely on nominal print size. Measure the actual square edge with calipers to set the precise metric scale.

  10. Export calibration data to 3D software and frameworks

    realtime-calib supports exporting calibration data to multiple targets. The export process automatically applies the correct axis remapping and handedness for each target:

    • three.js / OpenGL: Y-up, right-handed
    • Blender / ROS: Z-up, right-handed
    • Unity: Y-up, left-handed
    • Unreal: Z-up, left-handed
    • Caliscope: Compatible TOML export that preserves native field semantics, ensuring compatibility with camera_array.toml pipelines.

  11. Apply empirical best practices for camera calibration capture

    To maximize calibration accuracy, follow these empirical rules-of-thumb for board selection and image capture. Note that these are practitioner-based recommendations rather than strictly evidence-based constants.

    Board & Hardware Specifications:

    • Board Type: Use ChArUco boards.
    • Grid Size: Typically 5×7 to 8×11 squares (use 9×7+ for wide-angle).
    • Square Size: 15–50 mm (larger squares allow for longer working distances).
    • Marker/Square Ratio: 0.6–0.75 (realtime-calib default is 0.75).
    • Dictionary: Use the smallest dictionary that provides the required number of markers (e.g., DICT_5X5_100).
    • Substrate: Use a rigid and flat material such as aluminum or glass.

    Capture Strategy:

    • Frame Fill: The board should fill 30–70% of the frame.
    • Marker Resolution: Minimum marker/square size should be ≥ 8–12 pixels.
    • Tilt/Orientation: Use at least 2 non-parallel orientations, with tilts up to ~45–60°.
    • Image Count: Capture approximately 15–25 images (theoretical minimum is 3).
    • Lighting: 300–1000 lux, even lighting with no glare.

    Technical Parameters & Quality:

    • Distortion Model: Use the classic 5-coefficient [k1, k2, p1, p2, k3]. Use the rational 8-coefficient model only for strong wide-angle lenses.
    • Reprojection Error:
      • < 0.3 px: Good
      • 0.3–1.0 px: Acceptable

      • 1.0 px: Investigate for issues

  12. Select the appropriate calibration board

    The choice of calibration board depends on the specific calibration goal:

    • Intrinsic Calibration: Use a ChArUco board. Its ArUco markers provide robustness against occlusion and partial views, while interpolated chessboard corners ensure subpixel accuracy. This is ideal when the board fills a significant portion of the frame for one camera at a time.
    • Extrinsic Calibration (Large Volumes): Use a plain ArUco marker. In large spaces (rooms/halls), ChArUco squares become too small to detect reliably. A single ArUco marker remains detectable at a distance and is more cost-effective for linking cameras across a volume.

    Crucial Step: Always measure the printed board with calipers and enter the actual physical size into the system to ensure correct metric scale.