DOCSEARCHbyAlgolia
Submit Docs

OpenCompass Documentation

repository·main·Indexed 27 days ago

https://github.com/open-compass/opencompass

A comprehensive evaluation platform for Large Language Models (LLMs) that provides algorithms and interfaces to assess NLP model quality and effectiveness. It supports a wide array of benchmarks including AIME2024, APPS, ARC Prize Public Evaluation, ATLAS, BABILong, BBH, BuySideFinBench, CaLM Lite, and C-Eval.

Tokens
119.2K
Snippets
269
Records
500
Agent score
92%

What's inside OpenCompass

  2. Overview of the MMMLU dataset

    main

    Multilingual Massive Multitask Language Understanding (MMMLU) is a benchmark for general knowledge across 57 categories, ranging from elementary to advanced professional subjects. It consists of the MMLU test set translated into 14 languages by professional human translators to ensure high accuracy, especially for low-resource languages.

    Supported locales include:

    • AR_XY (Arabic)
    • BN_BD (Bengali)
    • DE_DE (German)
    • ES_LA (Spanish)
    • FR_FR (French)
    • HI_IN (Hindi)
    • ID_ID (Indonesian)
    • IT_IT (Italian)
    • JA_JP (Japanese)
    • KO_KR (Korean)
    • PT_BR (Brazilian Portuguese)
    • SW_KE (Swahili)
    • YO_NG (Yoruba)
    • ZH_CH (Simplified Chinese)

  3. Overview of CompassBench v1.0 Evaluation Tasks

    main

    CompassBench v1.0 is a comprehensive evaluation suite designed to support OpenCompass annual leaderboards. It assesses models across multiple dimensions including Language, Knowledge, Creativity, Reasoning, Mathematics, Code, Long Context, and Agent capabilities.

    Evaluation Methodologies:

    • Hybrid Approach: Combines subjective and objective evaluation methods depending on the specific task.
    • Reasoning/Math/Code/Agent Tasks: Utilizes Few-shot + CoT (Chain of Thought) prompting.
    • Fill-in-the-blank: Uses Few-shot examples and output format constraints in the prompt to assist answer extraction.
    • Multiple Choice: Employs varied questioning styles for the same problem to reduce randomness.
    • Open-ended Questions: Uses multiple samplings for the same question and evaluates via multi-dimensional scoring.

    Note: OpenCompass optimizes post-processing for objective questions. If a model fails to follow instruction formatting required for answer extraction, it is treated as an incorrect answer.

  4. Overview of LiveCodeBench dataset

    main

    LiveCodeBench is a dataset designed for holistic and contamination-free evaluation of Large Language Model (LLM) coding capabilities. It continuously collects new problems from LeetCode, AtCoder, and CodeForces to ensure evaluation remains fresh. The dataset covers multiple code-related capabilities including:

    • Code Generation
    • Test Output Prediction
    • Code Execution
    • Self-repair

    It currently hosts approximately 400 high-quality coding problems published between May 2023 and March 2024.

    Origin Project: https://livecodebench.github.io/leaderboard.html

  7. Overview of NeedleBench V2 Dataset Configuration

    main

    NeedleBench V2 is a benchmark designed to evaluate Large Language Models (LLMs) on information retrieval and reasoning capabilities within long-context scenarios. It provides specialized task configurations for various context lengths, ranging from 4k to 1000k tokens.

    Available Length Configurations

    The configuration directory configs/datasets/needlebench_v2/ contains subdirectories for each supported context length:

    • needlebench_v2_4k
    • needlebench_v2_8k
    • needlebench_v2_32k
    • needlebench_v2_128k
    • needlebench_v2_200k
    • needlebench_v2_256k
    • needlebench_v2_1000k

    Each subdirectory contains the specific test task configuration files for that length.

  8. Overview of MGSM benchmark

    main

    MGSM (Multilingual Grade School Math) is a benchmark consisting of 250 grade-school math problems manually translated from the GSM8K dataset into ten typologically diverse languages. It is designed to evaluate the multilingual chain-of-thought reasoning capabilities of language models.

    Input Format

    Models are typically prompted to provide reasoning steps followed by a final answer on the last line in the format Answer: <integer>.

    Supported Languages

    The benchmark includes versions for various languages such as English (mgsm_en), German (mgsm_de), Spanish (mgsm_es), French (mgsm_fr), Japanese (mgsm_ja), Russian (mgsm_ru), Swahili (mgsm_sw), Telugu (mgsm_te), Thai (mgsm_th), and Chinese (mgsm_zh), among others.

  10. Overview of FLAMES dataset

    main
    FLAMES is a value alignment benchmark designed for Chinese Large Language Models (LLMs). It evaluates models on common harmlessness principles and a unique morality dimension that integrates specific Chinese values, such as harmony. The benchmark covers several dimensions including data protection, legality, morality (non-environmental friendly, disobeying social norms, and Chinese values), safety (non-anthropomorphism, physical harm, mental harm, and property safety), and fairness.

  11. Overview of CaLM Lite tasks and structure

    main

    CaLM Lite provides 92 causal evaluation tasks stored in the data/calm folder. The tasks are organized by causal ladder and scenario.

    Directory Structure Example:

    ├── calm
| ├── association
| ├── causal_discovery
| │ ├── abstract_reasoning
| │ │ ├── AR-B_CaLM-AR_CN.json
| │ │ └── AR-B_CaLM-AR_EN.json
| │ └── ...
| └── ...
└── ...

    Dataset Composition:

    • Size: ~9,200 samples (compared to ~126,000 in the full CaLM).
    • Question Types: Includes Binary classification, Choice selection, and Probability calculation.
    • Modes: Natural, Symbolic, and Mathematical.

  12. Overview of SeedBench dataset

    main

    SeedBench is a multi-task benchmark designed to evaluate Large Language Models (LLMs) in the field of seed science, specifically focusing on seed breeding. It assesses models across three core stages:

    • Gene Information Retrieval
    • Gene Function and Regulation Analysis
    • Variety Breeding with Agronomic Trait Optimization

    The dataset is bilingual (English/Chinese) and features expert-validated questions targeting rice breeding, with plans to expand to maize, soybean, and wheat.