DOCSEARCHbyAlgolia
Submit Docs

MLE-bench

repository·main·Indexed 23 days ago

https://github.com/openai/mle-bench

A benchmark for evaluating Machine Learning (ML) agents on Machine Learning Engineering tasks. It features a dataset of 75 Kaggle competitions, evaluation logic, and a leaderboard. The toolkit includes utilities for dataset preparation, batch and individual grading of submissions, rule violation detection using gpt4o-mini, and plagiarism detection via the Dolos library. It provides a dedicated Docker image (mlebench-env) for agent environments and supports a 'Lite' evaluation split consisting of 22 low-complexity competitions.

Tokens
85.6K
Snippets
193
Records
597
Agent score
82%

What's inside mle-bench

  3. Aptos 2019 Blindness Detection: Task and Metric

    main

    The goal is to create a classifier to predict the severity of diabetic retinopathy from retina images.

    Metric: The competition uses Quadratic weighted kappa, which measures the agreement between human ratings and predicted scores.

    • The metric ranges from 0 (random agreement) to 1 (complete agreement), and can be below 0 if agreement is worse than chance.
    • Ratings are on a scale of 0 to 4.
    • The calculation involves constructing an $N \times N$ histogram matrix $O$ of observed ratings and an $N \times N$ matrix of expected ratings $E$ (calculated as the normalized outer product of the raters' histogram vectors).

  7. RSNA-MICCAI Brain Tumor Radiogenomic Classification Task Details

    main

    The goal is to predict the genetic subtype of glioblastoma by using MRI (magnetic resonance imaging) scans to detect the presence of MGMT promoter methylation.

    Metric: The performance is evaluated using the Area under the ROC curve (AUC) between the predicted probability and the observed target.

  8. Understand the NFL Player Contact Detection dataset structure

    main

    The dataset consists of video files, contact labels, baseline helmet detections, player tracking data, and video metadata.

    Video Files

    • [train/test] mp4: Each play includes three views: Sideline and Endzone (frame-synced) and All29 (not guaranteed to be time-synced with the others).
    • Frame Rate: 59.94 Hz.
    • Snap Timing: The moment of snap occurs exactly 5 seconds into the video.

    Label and Submission Formats

    • train_labels.csv: Contains contact events for player pairs or player-to-ground contacts at 0.1s intervals.
    • sample_submission.csv: Requires a binary prediction for contact.

    Supplemental Data

    • [train/test]_baseline_helmets.csv: Imperfect baseline predictions for helmet bounding boxes and player assignments for Sideline and Endzone views.
    • [train/test]_player_tracking.csv: High-frequency (10Hz) sensor data providing player positions, speed, and orientation.
    • [train/test]_video_metadata.csv: Metadata used to sync video files with player tracking data.

  10. Text Normalization Challenge: Task and Metric

    main

    The goal is to create a model that normalizes text by converting written expressions into forms suitable for text-to-speech systems (e.g., converting "$3.16" into "three dollars, sixteen cents").

    Metric: Prediction accuracy is measured as the total percent of correct tokens. A prediction is only considered correct if the predicted string and the actual string match exactly.