DOCSEARCHbyAlgolia
Submit Docs

norfair

repository·master·Indexed 25 days ago

https://github.com/tryolabs/norfair

A lightweight Python library for adding real-time multi-object tracking to any detector. Norfair converts detector outputs into coordinate-based detections and supports camera movement estimation via translation and homographies to stabilize tracking. It can be integrated with various detectors including YOLO, Detectron2, MMDetection, AlphaPose, and OpenPose, and supports Re-identification (ReID) using appearance embeddings to maintain tracks during occlusions.

Tokens
11.4K
Snippets
24
Records
84
Agent score
83%

What's inside norfair

  1. Overview of Norfair tracking capabilities

    master
    Norfair is a modular object tracking library designed to be easily integrated with various state-of-the-art (SOTA) object detectors. Unlike trackers that are hardcoded to specific detection formats (like bounding boxes), Norfair can work with any detector by supporting a variable number of points per detection. It also allows for heavy customization through user-defined distance functions.

  4. Track multiple classes with a single Tracker instance

    master
    The keypoints and bounding boxes demo demonstrates Norfair's capability to use a single Tracker instance to manage and track objects belonging to different classes simultaneously. It also showcases the ability to visualize both keypoints and bounding boxes for various object types within the same tracking session.

  5. Use OpenPose demo to extrapolate detections through skipped frames

    master

    The OpenPose demo demonstrates how to speed up inference by making a detector skip frames and using Norfair to extrapolate detections during those skipped frames.

    In this specific implementation, the detector skips 1 out of every 2 frames, which can make video processing up to 2 times faster because the computational overhead of Norfair is negligible compared to the cost of deep neural network inference.

  6. Understand ReID logic in Norfair

    master

    Re-identification (ReID) in Norfair is used when spatial distance matching fails (specifically when hit_counter is 0).

    Workflow:

    1. When spatial matching fails, trackers with hit_counter <= 0 attempt to calculate ReID distances.
    2. This continues until reid_hit_counter reaches 0, at which point the tracker is removed.
    3. If a ReID distance match is found, the trackers are merged.

    In the provided demo, the embedding is the color of the box (with noise), and the distance function is a histogram comparison between embeddings from unmatched_init_trackers and dead_objects against those from matched_not_init_trackers.

  7. Run the SAHI and Norfair detection and tracking demo

    master

    This demo demonstrates how to combine Norfair with SAHI (Slicing Aided Hyper Inference) and YOLOv5x to perform detection and tracking on small objects.

    To run the demo, you must use the provided Docker environment:

    1. Build and start the Docker container using the ./run.sh script.
    2. Inside the container, execute the demo script by passing a video file as an argument: python demo.py <video>.mp4.

    The process will generate an output video named output.mp4 containing the tracking results.

  8. Add tracking to a detector with Norfair

    master

    To use Norfair, you need to convert your detector's output into a list of norfair.Detection objects. Norfair then uses a Tracker to match these detections over time using a distance function (e.g., "euclidean").

    Basic workflow:

    1. Initialize a Video object for frame iteration.
    2. Initialize a Tracker with a distance_function and distance_threshold.
    3. In a loop, pass detections to tracker.update().
    4. Use draw_tracked_objects to visualize the results.
    import cv2
import numpy as np
from detectron2.config import get_cfg
from detectron2.engine import DefaultPredictor

from norfair import Detection, Tracker, Video, draw_tracked_objects

# Set up Detectron2 object detector
cfg = get_cfg()
cfg.merge_from_file("demos/faster_rcnn_R_50_FPN_3x.yaml")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5
cfg.MODEL.WEIGHTS = "detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x/137849600/model_final_f10217.pkl"
detector = DefaultPredictor(cfg)

# Norfair
video = Video(input_path="video.mp4")
tracker = Tracker(distance_function="euclidean", distance_threshold=20)

for frame in video:
    detections = detector(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
    detections = [Detection(p) for p in detections['instances'].pred_boxes.get_centers().cpu().numpy()]
    tracked_objects = tracker.update(detections=detections)
    draw_tracked_objects(frame, tracked_objects)
    video.write(frame)

  9. Estimate camera movement in Norfair

    master

    Norfair can estimate camera movement to stabilize object tracking and calculate trajectories in a fixed reference frame. This is particularly useful when camera motion causes erratic apparent object movement.

    There are two primary methods for estimating movement:

    1. Translation: Works for camera pans and tilts. It helps maintain object IDs when the camera moves horizontally or vertically.
    2. Homographies: A more robust method that works with any camera movement, including pan, tilt, rotation, movement in any direction, and zoom.

    Note: Estimation works best with a static background. Highly chaotic scenes with significant movement may reduce accuracy, though incorrect estimation will not negatively impact the tracking itself.

  10. Run the Moving Camera Demo

    master

    To run the moving camera demonstration, follow these steps:

    1. Build and run the Docker container using the provided script:
      ./run_gpu.sh
    2. Copy your target video file into the src folder.
    3. Inside the container, execute the demo script with your video file:
      python demo.py <video>.mp4

    To see all available configuration options, run:

    python demo.py --help
    ./run_gpu.sh
# After copying video to src/
python demo.py <video>.mp4

  11. Run the MMDetection tracking example using Docker

    master

    To run the MMDetection tracking demo, follow these steps:

    1. Build and start the GPU-enabled Docker container using the provided script:
      ./run_gpu.sh
    2. Place the video file you wish to process into the src folder.
    3. Inside the running container, execute the demo script by passing your video file as an argument:
      python demo.py <video>.mp4

    You can view all available command-line options by running python demo.py --help inside the container.

    ./run_gpu.sh
python demo.py <video>.mp4

  12. Run the ReID Demo using Docker

    master

    To run the Re-identification (ReID) demonstration, which shows how Norfair uses appearance embeddings to maintain tracks during occlusions, follow these steps:

    1. Build and run the Docker container using the provided script: ./run.sh.
    2. Inside the container, execute the demo script: python demo.py.

    This process generates two video files:

    • demo.avi: The original simulation video.
    • output.mp4: The result of the tracking with ReID applied.
    ./run.sh
# Inside the container:
python demo.py