DOCSEARCHbyAlgolia
Submit Docs

motmetrics

repository·develop·Indexed 23 days ago

https://github.com/cheind/py-motmetrics

A Python toolkit for evaluating Multiple Object Tracking (MOT) results, providing implementations for CLEAR MOT, Identity, and HOTA metrics aligned with the MOTChallenge benchmark. It includes high-level functions like evaluate_motchallenge, the MOTAccumulator for tracking data, and CLI tools for evaluating MOTChallenge and UA-DETRAC datasets. The library supports Python versions 3.8 through 3.14 and provides utilities for computing IoU and Euclidean distance matrices.

Tokens
5.6K
Snippets
3
Records
43
Agent score
81%

What's inside motmetrics

  1. Advanced components for custom evaluation

    develop

    For more granular control over the evaluation process, use these lower-level components:

    • mm.MOTAccumulator: Stores frame-level matching events.
    • mm.distances: Contains distance helpers like IoU and Euclidean matrices.
    • mm.io.loadtxt(..., fmt="auto"): Detects MOTChallenge, VATIC, and UA-DETRAC formats.
    • mm.metrics.create(): Returns a MetricsHost for custom metric selection.
    • mm.utils.compare_to_groundtruth: Compares loaded dataframes directly.
    • mm.utils.compare_to_groundtruth_reweighting: Supports custom HOTA-style multi-threshold workflows.

  2. Quick Start: Evaluate MOTChallenge results

    develop

    Use mm.evaluate_motchallenge to compute metrics for MOTChallenge-style text files or folders. The function automatically detects formats like MOTChallenge text, VATIC text, and UA-DETRAC .mat/.xml files.

    If the format is ambiguous, you can specify it explicitly using mm.io.Format.

    To evaluate a folder, ensure the following layout:

    gt_root/<SEQUENCE>/gt/gt.txt
preds_root/<SEQUENCE>.txt

  4. How metric dependencies work in MetricsHost

    develop

    The MetricsHost manages a directed acyclic graph (DAG) of metric dependencies. When you request a metric via .compute(), the host automatically identifies and computes all required dependency metrics first.

    Dependency Resolution:

    • Manual: You explicitly provide a list of strings to the deps parameter during registration.
    • Automatic: If deps='auto' is used, the host inspects the function signature. It assumes the first argument is the dataframe and subsequent arguments are the dependencies. The names of the arguments in the function signature must match the names of the registered dependency metrics.

    Example of Automatic Dependency:

    # 'num_matches' is registered first
mh.register(num_matches)

# 'num_detections' depends on 'num_matches'
# The argument name 'num_matches' must match the registered name
def num_detections(df, num_matches):
    return num_matches + ... 

mh.register(num_detections, deps='auto')

  5. Understand MOTAccumulator event types

    develop

    The accumulator generates several types of events based on the matching algorithm. These are stored in the .events property as a pandas DataFrame.

    Event Types:

    • MATCH: A match between an object and hypothesis was found.
    • SWITCH: A match was found, but the hypothesis ID differs from the previous assignment.
    • MISS: No match was found for an object.
    • FP: No match was found for a hypothesis (False Positive/Spurious detection).
    • RAW: Raw input events (used for internal tracking).
    • TRANSFER: A match was found, but the object ID differs from the previous assignment.
    • ASCEND: A match was found, but the hypothesis ID is new (sub-category of SWITCH).
    • MIGRATE: A match was found, but the object ID is new (sub-category of TRANSFER).

  6. Use MOTAccumulator to accumulate tracking metrics

    develop

    The MOTAccumulator class is the primary interface for managing tracking events frame-by-frame. You provide object IDs, hypothesis IDs, and a distance matrix for each frame, and the accumulator generates events like MATCH, SWITCH, MISS, and FP. Once all frames are processed, you can use metrics.summarize to compute overall tracking statistics.

    Key Workflow:

    1. Initialize MOTAccumulator.
    2. Call .update() for every frame with the current detections and ground truth.
    3. Use metrics.summarize(acc) to get the summary.
    4. Render the summary using io.render_summary(summary).

  7. Use the evaluateTracking CLI to compute MOT metrics

    develop

    The evaluateTracking.py script is a command-line tool designed to compute Multiple Object Tracking (MOT) metrics using ground-truth data following the MOTChallenge format.

    Data Requirements

    To use this tool, your data must follow this specific structure:

    Ground Truth Layout:

    <GT_ROOT>/<SEQUENCE_1>/gt/gt.txt
<GT_ROOT>/<SEQUENCE_2>/gt/gt.txt

    Test Data Layout:

    <TEST_ROOT>/<SEQUENCE_1>.txt
<TEST_ROOT>/<SEQUENCE_2>.txt

    Seqmap File: A text file containing the names of the sequences to be evaluated, one per line. Lines starting with #, empty lines, or the word name are ignored.

    Command Line Usage

    Run the script by providing the ground truth directory, the test results directory, and the seqmap file:

    python evaluateTracking.py <GT_ROOT> <TEST_ROOT> <SEQMAP_FILE> [options]

  8. Note on MOTP calculation differences

    develop

    When comparing py-motmetrics results to the official MOTChallenge devkit, note that py-motmetrics reports MOTP as the average distance, whereas MOTChallenge reports it as a percentage.

    To convert py-motmetrics MOTP to the MOTChallenge-style percentage, use the formula: (1 - MOTP) * 100.

  10. Evaluate MOTChallenge results via CLI

    develop

    You can run the evaluator directly from the command line by providing the paths to the ground truth root and predictions root.

    python -m motmetrics.apps.eval_motchallenge path/to/gt_root path/to/preds_root

  11. Register a new metric with MetricsHost

    develop

    To add a custom metric to a MetricsHost instance, use the register method.

    Arguments:

    • fnc: The function to compute the metric. The first argument must be the dataframe/accumulator. Subsequent arguments are the results of its dependencies.
    • deps: (Optional) A list of metric names that this metric depends on. If set to 'auto', the host attempts to deduce dependencies from the function's argument names.
    • name: (Optional) A unique identifier for the metric. Defaults to the function name.
    • helpstr: (Optional) A description of the metric.
    • formatter: (Optional) A string formatting function (e.g., '{:.2%}'.format) used when rendering results.
    • fnc_m: (Optional) A function used to merge results when computing overall metrics across multiple datasets. It must accept the list of partial results and the results of its dependencies (deps_m).
    • deps_m: (Optional) Dependencies for the merge function fnc_m.

  12. Load Vatic text format with load_vatictxt()

    develop
    Use load_vatictxt to load Vatic CSV text files. The function parses variable attributes present in the file and converts them into boolean columns. The returned pandas.DataFrame is indexed by ('FrameId', 'Id') and includes columns for X, Y, Width, Height, Lost, Occluded, Generated, ClassId, and any discovered attributes.