Mixture-of-Recursions (MoR)

BasqueGLUE is a Natural Language Understanding (NLU) benchmark for the Basque language. In this library, the basque-glue tag identifies the first version of the implementation, which calls all subtasks without averaging the results.

Supported tasks include:

• bhtc_v2: Topic classification of news extracts (12 categories).
• bec2016eu: Sentiment analysis on tweets regarding the 2016 Basque elections campaign.
• vaxx_stance: Stance detection on tweets related to the anti-vaccine movement.
• qnlieu: Q&A NLI (similar to glue/qnli).
• wiceu: Word-in-Context (similar to super_glue/wic).
• epec_koref_bin: Correference detection (similar to super_glue/wsc).

The Language Model Evaluation Harness provides a unified framework for testing generative language models across a wide variety of academic benchmarks.

Key Capabilities:

• Benchmark Support: Over 60 standard academic benchmarks with hundreds of subtasks.
• Model Compatibility: Supports models via transformers (including quantization via AutoGPTQ), GPT-NeoX, and Megatron-DeepSpeed. It also supports evaluation on adapters (e.g., LoRA via HuggingFace peft).
• Inference Optimization: Supports fast, memory-efficient inference using vLLM.
• API Support: Supports commercial APIs like OpenAI and TextSynth.
• Customization: Easy support for custom prompts, evaluation metrics, and local benchmarks.
• Reproducibility: Uses publicly available prompts to ensure comparability across research papers.

QA4MRE (Question Answering for Machine Reading Evaluation) is a challenge dataset used for evaluating Machine Reading systems through Question Answering and Reading Comprehension tests. The dataset focuses on extracting knowledge from large volumes of text to answer questions.

Available task groups and specific tasks include:

• Group: qa4mre
• Tasks:
  • qa4mre_2011
  • qa4mre_2012
  • qa4mre_2013

The noticia task is a Spanish clickbait article summarization benchmark. It uses the NoticIA dataset, which consists of 850 Spanish news articles with clickbait headlines paired with human-written, single-sentence generative summarizations. The goal is to evaluate a model's ability to infer information from clickbait and generate high-quality summaries.

Metrics used:

• Rouge1 score
• Average Summary Length

WSC273 (Winograd Schema Challenge 273) is a benchmark consisting of 273 Winograd schemas. A Winograd schema is a pair of sentences that differ by only one or two words and contain an ambiguity that requires world knowledge and reasoning to resolve.

This specific implementation uses partial evaluation as described by Trinh & Le in Simple Method for Commonsense Reasoning (2018).

MedConceptsQA is an open-source benchmark designed for evaluating Large Language Models (LLMs) on medical concept question answering. It tests the ability of models to interpret medical codes and distinguish between various medical concepts across three difficulty levels: easy, medium, and hard.

The benchmark covers three primary medical vocabularies:

• Diagnoses
• Procedures
• Drugs