DOCSEARCHbyAlgolia
Submit Docs

LM Evaluation Harness

repository·main·Indexed 11 days ago

https://github.com/eleutherai/lm-evaluation-harness

A unified framework for testing and benchmarking generative language models across hundreds of academic tasks. Version 0.4.13.dev0 serves as the backend for the Hugging Face Open LLM Leaderboard and supports diverse benchmarks including ACLUE, ACPBench, Arabic EXAMS, IrokoBench (AfriMGSM, AfriMMLU, AfriXNLI), and AfroBench (ADR, AfriSenti, Belebele).

Tokens
116.8K
Snippets
279
Records
535
Agent score
95%

What's inside LM Evaluation Harness

  1. Overview of the EgyHellaSwag benchmark

    main

    EgyHellaSwag is a benchmark designed to evaluate commonsense reasoning specifically in Egyptian Arabic. It is a translated version of the HellaSwag benchmark, utilizing multiple-choice sentence completion tasks. The dataset was generated using the google/gemma-3-27b-it model.

    Dataset Details:

  3. Overview of MultiMedQA (multiple-choice subset) benchmark

    main

    MultiMedQA is a benchmark designed to evaluate how well Large Language Models encode clinical knowledge. This specific subset focuses on multiple-choice question answering across four primary datasets:

    • PubMedQA: A closed-domain Q&A task consisting of 1,000 expert-labeled pairs where a question and a PubMed abstract are provided as context. The required answers are yes, maybe, or no.
    • MedQA: US Medical License Exam (USMLE) questions. While it contains 4 or 5 possible answers, the 4-option questions are typically used for evaluation.
    • MedMCQA: Multiple-choice questions from Indian medical entrance examinations, featuring 4 options per question.
    • MMLU (Medical domains): A subset of the Massive Multitask Language Understanding benchmark focusing on 6 medical domains:
      • Anatomy
      • Clinical Knowledge
      • College Medicine
      • Medical Genetics
      • Professional Medicine
      • College Biology

    Note: This subset excludes short-form and long-form Q&A tasks (like LiveQA, MedicationQA, and HealthSearchQA) because they typically require expert evaluation rather than automated scoring.

  4. Overview of the GroundCocoa benchmark

    main
    GroundCocoa is a benchmark designed to evaluate compositional and conditional reasoning in Large Language Models (LLMs). It uses a lexically diverse setup centered around a real-world flight booking problem, where models must align detailed user preferences with available flight options in a multiple-choice format. The benchmark is intended to identify areas where LLMs falter despite high performance on standard benchmarks.

  5. Overview of Histoires Morales dataset

    main

    Histoires Morales is a French dataset designed for assessing moral model alignment. It consists of 12,000 narratives that describe social norms, situations, and intentions, followed by two paths: a normative path (action and consequence that follow the norm) and a norm-divergent path (action and consequence that diverge from the norm). The dataset is a French adaptation of the Moral Stories dataset, refined through manual annotations.

    Dataset Structure:

    1. Norm: Social conduct guideline.
    2. Situation: Setting and participants.
    3. Intention: The actor's goal.
    4. Normative action: Action fulfilling intention while observing the norm.
    5. Normative consequence: Effect of the normative action.
    6. Divergent action: Action fulfilling intention but diverging from the norm.
    7. Divergent consequence: Effect of the divergent action.

    Resources:

  6. Overview of TokSuite Benchmark

    main

    TokSuite is a benchmark designed to measure the impact of tokenizer choice on language model behavior. It consists of seven main dataset groups, each containing multiple-choice completion questions. The benchmark evaluates how real-world perturbations (like typos, formatting changes, or script switching) affect model performance compared to a 'canonical' (natural) form.

    Benchmark Groups:

    • Parallel Multilingual Benchmark: Includes English canonical subsets and translations into Turkish (tur_Latn), Chinese (zho_Hans), Italian (ita_Latn), and Farsi/Persian (pes_Arab).
    • STEM Benchmark: Covers STEM subjects with various formatting perturbations.
    • MATH Benchmark: Arithmetic questions across multiple languages.

    Each group contains numerous specific tasks (e.g., toksuite_english_lowercase, toksuite_turkish_typographical_errors, toksuite_math_latex) that target specific linguistic or orthographic challenges.

  7. Overview of KBL (Korean Benchmark for Legal Language Understanding)

    main

    KBL is a benchmark designed to assess the Korean legal language understanding of Large Language Models (LLMs). It covers three main areas:

    1. Legal Knowledge Tasks: 7 tasks (510 examples) evaluating specific legal knowledge.
    2. Legal Reasoning Tasks: 4 tasks (288 examples) evaluating reasoning capabilities.
    3. Korean Bar Exam: 53 tasks (2,510 examples) across 4 domains (Civil, Criminal, Public, and Professional Responsibility).

    The benchmark evaluates models in both closed-book settings and Retrieval-Augmented Generation (RAG) settings using a corpus of Korean statutes and precedents.

  8. Overview of BasqueBench tasks

    main

    BasqueBench is a benchmark designed to evaluate a language model's ability to understand and generate Basque text. It includes both newly developed datasets and existing open datasets.

    New BasqueBench Datasets

    • Question Answering: ARC_eu, PIQA_eu
    • Math: MGSM_eu
    • Paraphrasing: PAWS_eu
    • Natural Language Inference: WNLI_eu
    • Commonsense Reasoning: XCOPA_eu

    Existing Datasets included in BasqueBench

    • Reading Comprehension: Belebele_eu, EusReading
    • Question Answering: EusExams, EusProficiency, EusTrivia
    • Natural Language Inference: QNLIeu, XNLIeu
    • Commonsense Reasoning: XStoryCloze_eu
    • Translation: FLORES_eu
    • Paraphrasing: PAWS_eu (Note: listed in both categories in source)

  9. Overview of CatalanBench

    main
    CatalanBench is a benchmark designed to evaluate the ability of language models to understand and generate Catalan text. It consists of a combination of existing open datasets and datasets developed exclusively for this benchmark. The benchmark covers various categories including Question Answering, Math, Paraphrasing, Commonsense Reasoning, Summarization, Linguistic Acceptability, Translation, and Natural Language Inference.

  10. Overview of TurBLiMP tasks

    main

    TurBLiMP is a Turkish benchmark consisting of linguistic minimal pairs designed to evaluate monolingual and multilingual language models across 16 core grammatical phenomena.

    In the lm-evaluation-harness, you can run the entire suite using the group turblimp_core, or select specific subtasks.

    Metrics Note: Due to differences in length normalization compared to the original implementation, this harness provides two metrics:

    • acc: Accuracy based on unnormalized log-probability.
    • acc_norm: Accuracy where sentence log-probability is normalized by the number of bytes.

  11. Overview of SLR-Bench

    main

    SLR-Bench is a benchmark designed for scalable logical reasoning in language models. The tasks involve generating Prolog rules that must correctly classify trains based on their compositions.

    Key components include:

    • Dataset: Hosted on Hugging Face at AIML-TUDA/SLR-Bench.
    • Verifier: Uses a symbolic verifier (AIML-TUDA/VerifiableRewardsForScalableLogicalReasoning) to evaluate the correctness of generated rules.
    • Research Context: Based on the paper "SLR: Automated Synthesis for Scalable Logical Reasoning" (arXiv:2506.15787).

  12. Overview of GalicianBench

    main
    GalicianBench is a benchmark designed to evaluate the ability of language models to understand and generate Galician text. It utilizes a combination of existing open datasets and datasets developed specifically for this benchmark. It covers various linguistic categories including Reading Comprehension, Linguistic Acceptability, Math, Paraphrasing, Question Answering, Summarization, Truthfulness, NLI, and Commonsense Reasoning.