repository·main·Indexed 41 days ago
https://github.com/mlcommons/medperfAn open benchmarking platform for medical artificial intelligence designed for Federated Evaluation. MedPerf enables the evaluation of AI models across multiple sites without moving sensitive medical data using core entities such as Benchmarks, MLCubes (containers), Datasets, Registrations, and Results. It provides structured templates for creating Data Preparator, Metrics, and Model MLCubes to standardize data preparation, performance measurement, and model inference.
The BraTS MLCubes folder provides a suite of MedPerf MLCubes (Machine Learning Cubes) designed for the Brain Tumor Segmentation (BraTS) challenge. These examples demonstrate a complete workflow from data preparation to model inference and metric calculation.
Note: This specific example set is currently marked as incompatible with the latest MedPerf version and is awaiting a fix.
The MedPerf tutorials provide a simulated benchmarking workflow using a chest X-ray thoracic disease detection task. They are designed to teach the fundamentals of the MedPerf client by interacting with a temporary local MedPerf server. These tutorials serve as general guidance for using the MedPerf client in real-world scenarios.
To use these tutorials, you must first complete the general tutorial environment setup. The tutorials are categorized by user role:
The Chest X-Ray Benchmark is an example implementation within MedPerf that uses MLCubes to perform chest X-Ray pathology detection.
Note: This example is currently not compatible with the latest MedPerf version. This will be fixed soon.
The benchmark is composed of three specialized MLCubes:
chexpert_prep: A Data Preparator MLCube used for preprocessing and standardizing input data.xrv_densenet: A pretrained Densenet model designed for chest X-Ray pathology detection.metrics: An Evaluator MLCube used to compute metric results based on model predictions.SurgMLCube is an example suite for benchmarking machine learning models on surgical videos. It is composed of three distinct MLCubes that represent different stages of a benchmarking pipeline: data preparation, model inference, and evaluation.
Note: This example is currently not compatible with the latest MedPerf version. Compatibility fixes are pending.
MedPerf is an open benchmarking platform for medical artificial intelligence that utilizes Federated Evaluation. The repository contains the following core components:
server directory.cli directory.examples directory.MedPerf is an open-source framework designed for benchmarking medical Machine Learning (ML) models. It utilizes a method called Federated Evaluation, which allows medical ML models to be securely distributed to multiple global facilities for testing.
This approach prioritizes patient privacy by ensuring that no patient data is shared across facilities during the evaluation process. The workflow involves:
This enables developers to identify bias and generalizability issues while allowing healthcare stakeholders (like hospitals) to validate clinical efficacy without regulatory or legal risks associated with data transfer.
The mlcube/workspace/parameters.yaml file allows you to define key-value pairs that can be modified to adjust model behavior (e.g., batch_size or patching settings) without rebuilding the container image.
In MedPerf, you can register the same MLCube image multiple times using different parameters.yaml files to compare different configurations (e.g., comparing a model with patching: true against one with patching: false).
# Example parameters.yaml
# Use these to adjust model behavior externally
uppercase: false
patching: trueMedPerf is built around several key abstractions used to prepare datasets and execute benchmarks. Understanding these entities is essential for managing the lifecycle of a clinical AI performance measurement:
To avoid baking large files (like model weights or hyperparameters) directly into your Docker/Singularity image, you can use MedPerf's mounting mechanism. When submitting container metadata to the MedPerf server, you can provide an additional_files.tar.gz (via a URL) and a parameters.yaml file.
At runtime, MedPerf mounts these into the container at the following absolute paths:
additional_files.tar.gz are uncompressed and available in /mlcommons/volumes/additional_files.parameters.yaml are available at /mlcommons/volumes/parameters/parameters.yaml.MedPerf is designed to facilitate neutral governance, which enables full trust and transparency between participating parties such as AI vendors, data providers, and regulatory bodies.
This governance is managed by a benchmark committee, which acts as the overseeing body for a specific benchmark, ensuring the evaluation process is efficient, human-supervised, and unbiased.
MedPerf facilitates federated benchmarking through three distinct roles, each with specific responsibilities:
The typical workflow involves: Benchmark Owner defines benchmark $\rightarrow$ Model Owner registers model $\rightarrow$ Model Owner requests participation $\rightarrow$ Data Owner registers dataset $\rightarrow$ Data Owner runs preparation $\rightarrow$ Data Owner requests participation $\rightarrow$ Benchmark Owner approves $\rightarrow$ Data Owner runs benchmark $\rightarrow$ Data Owner submits results $\rightarrow$ Benchmark Owner views results.