repository·main·Indexed 27 days ago
https://github.com/simplescaling/s1A unified framework for evaluating generative language models against academic benchmarks, serving as the backend for the Hugging Face Open LLM Leaderboard. It supports a wide array of tasks including AGIEval, GSM8K, IrokoBench (AfriMMLU, AfriMGSM, AfriXNLI), ACLUE, and Arabic EXAMS, providing guides for using the public interface, extending models via APIs, and adding or configuring new evaluation tasks.
lm_eval library as a collection of various tasks and groups.RACE (Large-scale ReAding Comprehension Dataset From Examinations) is a reading comprehension dataset consisting of over 28,000 passages and nearly 100,000 questions. The dataset is derived from English examinations for middle and high school students in China and is used for training and testing machine comprehension models.
Task Name in lm_eval: race
arc_mt task implementation provides support for machine-translated ARC (AI2 Reasoning Challenge) evaluation tasks. This is designed to improve evaluation support across multiple languages beyond English. Initial datasets include several European languages, with plans for future expansion.WSC273 (Winograd Schema Challenge 273) is a benchmark task designed to evaluate commonsense reasoning. It consists of 273 Winograd schemas—pairs of sentences that differ by only one or two words and contain an ambiguity that requires world knowledge to resolve.
This specific implementation in the lm_eval library uses partial evaluation as described by Trinh & Le in Simple Method for Commonsense Reasoning (2018) (https://arxiv.org/abs/1806.0).
QA4MRE (Question Answering for Machine Reading Evaluation) is a challenge dataset from CLEF 2011-2013 designed to evaluate Machine Reading systems through Question Answering and Reading Comprehension. The tasks focus on extracting knowledge from large volumes of text to answer questions.
Available task groups and specific tasks include:
qa4mreqa4mre_2011qa4mre_2012qa4mre_2013MC-TACO is a dataset consisting of 13,000 question-answer pairs designed to evaluate temporal commonsense comprehension. It covers five specific temporal properties:
To run this task in the lm_eval harness, use the task name: mc_taco.
EusTrivia is a benchmark consisting of 1,715 multiple-choice trivia questions used for evaluating language models in Basque. The dataset covers five main knowledge areas:
Questions range from elementary level (grades 3-6) to challenging levels, with a significant focus on the Basque Country, its language, and culture.
The inverse_scaling task group in lm_eval provides implementations of datasets from the 'Inverse Scaling: When Bigger Isn't Better' paper. These tasks are designed to evaluate scenarios where larger model scale may lead to worse performance due to factors like memorized sequence repetition, imitation of undesirable patterns, or distractor tasks.
Note: This is not the official implementation of the Inverse Scaling Prize; it is implemented by h-albert-lee with permission from the authors.
Paloma is a comprehensive benchmark designed to evaluate open language models across 585 distinct domains, ranging from niche artist communities to mental health forums on Reddit. It assesses model performance across a wide variety of specialized datasets.
PROST (Physical Reasoning about Objects Through Space and Time) is a dataset of 18,736 multiple-choice questions based on 14 manually curated templates. It covers 10 physical reasoning concepts and is designed to probe both causal and masked language models in a zero-shot setting.
Note: To adhere to the authors' intentions, this task is limited to zero-shot evaluation to ensure models are being tested on data they were not specifically trained to succeed on.