DOCSEARCHbyAlgolia
Submit Docs

TrackEval

repository·master·Indexed 23 days ago

https://github.com/jonathonluiten/trackeval

A high-performance Python library for evaluating object tracking algorithms. It provides a programmable API and command-line tools to compute various metric families, including HOTA, CLEARMOT, Identity, VACE, Track mAP, and J & F metrics. TrackEval supports multiple benchmarks such as MOTChallenge (MOT15, MOT16, MOT17, MOT20), KITTI, BDD-100k, RobMOTS, and Open-World Tracking (TAO-OW).

Tokens
4.4K
Snippets
8
Records
20
Agent score
78%

What's inside TrackEval

  2. Install and setup the RobMOTS evaluation environment

    master

    To use the official RobMOTS evaluation code, follow these steps to set up a Python 3.5+ environment, install dependencies, and download the necessary groundtruth and example data.

    # Download the TrackEval repo
git clone https://github.com/JonathonLuiten/TrackEval.git

# Move to repo folder
cd TrackEval

# Create a virtual env in the repo for evaluation
python3 -m venv ./venv

# Activate the virtual env
source venv/bin/activate

# Update pip to have the latest version of packages
pip install --upgrade pip

# Install the required packages
pip install -r requirements.txt

# Download the train gt data
wget https://omnomnom.vision.rwth-aachen.de/data/RobMOTS/train_gt.zip

# Unzip the train gt data you just downloaded.
unzip train_gt.zip

# Download the example tracker 
wget https://omnomnom.vision.rwth-aachen.de/data/RobMOTS/example_tracker.zip

# Unzip the example tracker you just downloaded.
unzip example_tracker.zip

# Run the evaluation on the provided example tracker on the train split (using 4 cores in parallel)
python scripts/run_rob_mots.py --ROBMOTS_SPLIT train --TRACKERS_TO_EVAL STP --USE_PARALLEL True --NUM_PARALLEL_CORES 4
    # Download the TrackEval repo
git clone https://github.com/JonathonLuiten/TrackEval.git

# Move to repo folder
cd TrackEval

# Create a virtual env in the repo for evaluation
python3 -m venv ./venv

# Activate the virtual env
source venv/bin/activate

# Update pip to have the latest version of packages
pip install --upgrade pip

# Install the required packages
pip install -r requirements.txt

# Download the train gt data
wget https://omnomnom.vision.rwth-aachen.de/data/RobMOTS/train_gt.zip

# Unzip the train gt data you just downloaded.
unzip train_gt.zip

# Download the example tracker 
wget https://omnomnom.vision.rwth-aachen.de/data/RobMOTS/example_tracker.zip

# Unzip the example tracker you just downloaded.
unzip example_tracker.zip

# Run the evaluation on the provided example tracker on the train split (using 4 cores in parallel)
python scripts/run_rob_mots.py --ROBMOTS_SPLIT train --TRACKERS_TO_EVAL STP --USE_PARALLEL True --NUM_PARALLEL_CORES 4

  3. Run TrackEval via terminal scripts

    master

    You can run evaluations using the provided scripts in the scripts/ directory. These scripts accept various arguments to specify benchmarks, trackers, and metrics.

    Example: Running CLEAR and Identity metrics on the MOT17 benchmark for the Lif_T tracker:

    python scripts/run_mot_challenge.py --BENCHMARK MOT17 --TRACKERS_TO_EVAL Lif_T --METRICS CLEAR Identity --USE_PARALLEL False --NUM_PARALLEL_CORES 1

  4. Evaluate on a custom benchmark

    master

    To include your own dataset and evaluate it as a custom benchmark <YourChallenge>, follow these two steps:

    1. Ground truth data preparation

    Prepare your sequences in TrackEval/data/gt/mot_challenge/<YourChallenge>/ using the following structure:

    .
|—— <SeqName01>
|   |—— gt
|   |   |—— gt.txt
|   |—— seqinfo.ini
|—— <SeqName02>
|   |—— ...

    2. Sequence file creation

    Create text files in the seqmaps folder containing the sequence names. The files should follow the pattern <YourChallenge>-<split>.txt (e.g., <YourChallenge>-train.txt, <YourChallenge>-test.txt, or <YourChallenge>-all.txt).

    Each file must start with the header name followed by the sequence names:

    name
<seqName1>
<seqName2>

    3. Execution

    Adjust scripts/run_mot_challenge.py and set BENCHMARK = <YourChallenge> to run the evaluation.

  6. Evaluate on the RobMOTS validation and test servers

    master

    Ground truth (GT) for the validation and test sets is not provided locally. Instead, you must use the live evaluation servers to upload your tracking results.

    • Validation Server: Allows infinite uploads.
    • Test Server: Limits trackers to a total of 4 uploads.

    Evaluation servers are located at: https://eval.vision.rwth-aachen.de/vision/.

    Before uploading, ensure your files follow the required format. You can download example upload files here:

  7. Run evaluation using run_mot_challenge.py

    master

    Execute the evaluation script located at scripts/run_mot_challenge.py. You can pass several arguments to customize the run:

    • --BENCHMARK: Name of the benchmark (e.g., MOT15, MOT16, MOT17, or MOT20). Default is MOT17.
    • --SPLIT_TO_EVAL: Data split to evaluate (e.g., train, test). Default is train.
    • --TRACKERS_TO_EVAL: List of tracker names to evaluate. Default is all trackers in the tracker folder.
    • --METRICS: List of metric families to compute (e.g., HOTA, CLEAR, Identity, VACE). Default is HOTA CLEAR Identity.
    • --USE_PARALLEL: Boolean to enable parallel evaluation on multiple cores. Default is False.
    • --NUM_PARALLEL_CORES: Number of cores to use if --USE_PARALLEL is True. Default is 8.
    python scripts/run_mot_challenge.py --BENCHMARK MOT17 --SPLIT_TO_EVAL train --TRACKERS_TO_EVAL MPNTrack --METRICS HOTA CLEAR Identity VACE --USE_PARALLEL False --NUM_PARALLEL_CORES 1

  8. Install TrackEval requirements

    master

    TrackEval requires numpy and scipy as minimum dependencies. Depending on your needs, you can install different sets of requirements:

    • Minimum requirements: For basic metric calculation.
    • Full requirements: Includes matplotlib for plotting, pycocotools and Pillow for segmentation datasets, opencv_python and scikit_image for J & F metrics, and pytest for testing.

    Use the following commands to install:

  9. Quickly evaluate on supported benchmarks using example data

    master

    To quickly test TrackEval, you can download ground-truth data and example trackers.

    1. Download the data: data.zip (~150mb).
    2. Extract the zip into the repository root folder so that paths follow the pattern: TrackEval/data/gt/....
    3. Run any script from the scripts/ directory without arguments. The code will automatically evaluate all trackers present in the file structure.

    Note on custom trackers: To evaluate your own results, copy your tracker files into a new folder at the same level as the example trackers (e.g., MPNTrack, CIWT, track_rcnn, etc.), maintaining the same internal file structure.

  10. Evaluate your own tracker

    master

    To evaluate a custom tracker, place your results in the folder TrackEval/data/trackers/rob_mots/train/ (or the relevant split folder). Create a subfolder named after your tracker (e.g., YOUR_TRACKER) and place your sequence .txt files inside it.

    Then, run the evaluation script using the --TRACKERS_TO_EVAL flag pointing to your folder name:

    python scripts/run_rob_mots.py --ROBMOTS_SPLIT train --TRACKERS_TO_EVAL YOUR_TRACKER --USE_PARALLEL True --NUM_PARALLEL_CORES 4
    python scripts/run_rob_mots.py --ROBMOTS_SPLIT train --TRACKERS_TO_EVAL YOUR_TRACKER --USE_PARALLEL True --NUM_PARALLEL_CORES 4

  11. Evaluate on a custom benchmark

    master

    To evaluate your own custom dataset and trackers, you have two options:

    1. Convert your data: Convert your dataset and trackers to match the format of an existing implemented benchmark (e.g., MOTChallenge format). Check the docs/ folder for specific format specifications.
    2. Custom dataset code: Write custom dataset code (requires more effort).

    Tip: When converting formats, it is often recommended to use the argument --DO_PREPROC False unless you specifically need to run preprocessing to remove distractor objects.

  12. Add a new or custom family of evaluation metrics to TrackEval

    master

    To implement a new metric or a custom family of metrics, follow these steps:

    1. Create the file: Place your metric code in trackeval/metrics/<your_metric>.py. It is recommended to use an existing metric as a template (e.g., trackeval/metrics/identity.py).
    2. Define the class: Your metric must be a class that inherits from trackeval.metrics._base_metric._BaseMetric.
    3. Initialize fields: Define an __init__ function that specifies the different fields (values) your metric will calculate. You can refer to trackeval/metrics/_base_metric.py for existing field types or define new ones.
    4. Implement core logic:
      • Implement eval_sequence(data_dict): This function calculates the metric for a single sequence and single class. It takes a data dictionary as input and returns a results dictionary.
    5. Implement aggregation logic:
      • combine_sequences: Defines how to combine metric field values over sequences.
      • combine_classes_class_averaged: Defines how to combine values over classes using class averaging.
      • combine_classes_det_averaged: Defines how to combine values over classes weighted by the number of detections.
      • Note: Using a helper like _compute_final_fields is recommended for convenience but not required.
    6. Register the metric: Add your new metric class to trackeval/metrics/init.py.
    7. Use the metric: Pass your metric class in a list to the evaluator (refer to scripts in the scripts/ directory for usage patterns).