DOCSEARCHbyAlgolia
Submit Docs

EvalScope Documentation

repository·main·Indexed 25 days ago

https://github.com/modelscope/evalscope

A lightweight LLM evaluation framework for model capability assessment, inference performance stress testing, and result visualization. It provides a unified interface for various benchmarks, model types, and evaluation backends, including support for Agent Evaluation Mode with pluggable strategies and tools, and an External Agent Bridge for evaluating agent CLIs. Features include a web dashboard for viewing predictions and metrics, and support for tasks like LongBench-Write and ToolBench-Static.

Tokens
306.6K
Snippets
831
Records
1.4K
Agent score
82%

What's inside EvalScope

  1. Overview of ArxivRollBench-Full benchmark

    main

    ArxivRollBench is a rolling benchmark designed to evaluate the ability of large language models (LLMs) to reason over recent scientific text from arXiv papers. It uses three task formats: sequencing, cloze, and next-fragment prediction (under the SCP framework).

    Key Details:

    • Task Type: Multiple-choice scientific text reasoning.
    • Input: Recent arXiv text fragments with four answer choices (A, B, C, or D).
    • Output: A single correct answer letter.
    • Domains: Computer Science, Quantitative Finance, Mathematics, Physics, Statistics, Quantitative Biology, Economics, and Electrical Engineering/System Science.
    • Snapshots: Includes 2024b, 2025a, and 2026a rolling snapshots to reduce contamination-related overestimation.
    • Metrics: Evaluated using accuracy (acc).
    • Dataset ID: liangzid/arxivrollbench-full (hosted on ModelScope).

  2. Overview of ACEBench Benchmark

    main

    ACEBench is a tool-use benchmark designed to evaluate Large Language Models (LLMs) on their ability to:

    • Select appropriate APIs.
    • Fill function arguments correctly.
    • Handle abnormal or edge-case requests.
    • Complete realistic agentic tasks.

    Key Details:

    • Benchmark Name: acebench
    • Dataset ID: evalscope/acebench
    • Task Type: Function calling and agentic tool use.
    • Subsets:
      • normal: Standard function calling tasks.
      • special: Handling abnormal requests/diagnostic cases.
      • agent: Complex, multi-step agentic tasks.
    • Metrics: acc, process_acc, and end_state_acc.

  3. Overview of MeasureBench Benchmark

    main

    measure_bench is a benchmark designed to evaluate Vision-Language Models (VLMs) on their ability to read values from measuring instruments. It includes 2,442 samples across two splits: real_world (1,272 samples) and synthetic_test (1,170 samples). The benchmark covers 26 instrument types (e.g., ammeters, clocks, thermometers) across 4 design categories: dial, digital, analog, and linear.

    Key Evaluation Details:

    • Task Type: Free-form Visual Question Answering.
    • Expected Output Format: Models must provide the answer on the last line in the format: Answer: <value> <unit>.
    • Metrics:
      • acc (all_correct): Both numeric value and unit are correct.
      • number_acc: Numeric value accuracy only.
      • unit_acc: Unit accuracy only.
    • Evaluators: Uses interval_matching (single range) and multi_interval_matching (for ambiguous cases like clock AM/PM).

  4. Overview of EvalScope framework

    main
    EvalScope is a comprehensive model evaluation and benchmarking framework designed for various model types, including Large Language Models (LLMs), Multimodal Models, Embedding Models, Reranker Models, CLIP Models, and AIGC Models (Text-to-Image/Video). It supports industry-recognized benchmarks (MMLU, CMMLU, C-Eval, GSM8K, etc.), performance stress testing, and integrates with the ms-swift training framework for end-to-end model development.

  5. Overview of WMT2024++ Benchmark

    main

    WMT2024++ is a machine translation benchmark based on the WMT 2024 news translation task. It evaluates translation quality from English into 54 different target languages using news domain text.

    Key Details:

    • Task Type: Machine Translation (English $\rightarrow$ Target Language).
    • Default Evaluation: 0-shot.
    • Metrics Supported: bleu, bert_score (using XLM-RoBERTa), and comet (using wmt22-comet-da).
    • Dataset ID: extraordinarylab/wmt24pp.
    • Subsets: Individual language pairs are identified by keys like en-zh_cn or en-de_de.

    Note: To use the comet metric, you must have the unbabel-comet package installed.

  6. Overview of OmniDocBench Benchmark

    main
    OmniDocBench is an evaluation benchmark designed for document parsing and understanding in real-world scenarios. It evaluates a model's ability to take a PDF page image as input and produce a parsed document structure in Markdown format. The benchmark covers 9 document types (including academic papers, financial reports, newspapers, textbooks, and handwritten notes), 4 layout types, and 3 language types across 1,355 PDF pages.

  7. Overview of IQuiz Benchmark

    main

    IQuiz is a Chinese benchmark designed to evaluate AI models on Intelligence Quotient (IQ) and Emotional Quotient (EQ). It uses multiple-choice questions to test logical reasoning, pattern recognition, and social-emotional understanding.

    Key Details:

    • Task Type: Multiple-Choice Question Answering (MCQ).
    • Language: Chinese.
    • Input Format: Question with multiple choice options.
    • Output Format: Selected answer with Chain-of-Thought (CoT) explanation.
    • Primary Metric: Accuracy (acc).
    • Subsets: IQ (logical reasoning) and EQ (emotional intelligence).
    • Default Setting: 0-shot evaluation on the test split.

  8. Overview of FRAMES Benchmark

    main

    FRAMES is a RAG (Retrieval-Augmented Generation) evaluation dataset designed to test long-context capabilities. It focuses on factuality, retrieval accuracy, and multi-hop reasoning using Wikipedia-sourced context documents.

    Key Properties:

    • Benchmark Name: frames
    • Dataset ID: iic/frames
    • Primary Metric: acc (Accuracy)
    • Evaluation Split: test
    • Default Shots: 0-shot
    • Task Type: RAG Evaluation / Long-Context QA

    Evaluation Details:

    • Supports both exact match (using normalized answer comparison) and LLM judge (for semantic matching).
    • Requires models to follow a specific response format: "Therefore, the answer is (answer here)".

  9. Overview of K2-Vendor-Verifier Benchmark

    main

    The k2_verifier benchmark is designed to verify whether third-party deployments of Kimi-K2 faithfully reproduce the official Moonshot AI API's tool-calling behavior. It replays official evaluation prompts against a vendor endpoint and compares the finish_reason and tool_call payloads against an official baseline.

    Key Details:

    • Benchmark Name: k2_verifier
    • Dataset ID: evalscope/K2VendorVerifier
    • Task Type: Vendor-deployment correctness check (tool calling)
    • Default Shots: 0-shot
    • Tags: Agent, FunctionCalling
    • Acceptance Threshold: A trigger_similarity ≥ 0.73 is considered a rough acceptance threshold.

  10. Overview of EvalScope features

    main

    EvalScope is a one-stop evaluation framework for large models with the following capabilities:

    • Comprehensive Benchmarks: Built-in support for MMLU, C-Eval, GSM8K, etc.
    • Multimodal & Multi-domain Support: Evaluates LLMs, VLMs, Embedding, Reranker, and AIGC models.
    • Multi-backend Integration: Integrates with OpenCompass, VLMEvalKit, RAGEval, and more.
    • Agent Evaluation Mode: Supports driving benchmarks like GSM8K, AIME, and SWE-bench Agentic within a controlled AgentLoop, featuring pluggable strategies, tools, and Docker sandboxes with full Agent Trace recording.
    • Inference Performance Testing: Tools for stress testing model services with metrics like TTFT and TPOT.
    • Interactive Reports: WebUI for multi-dimensional model comparison and detailed report viewing.
    • Arena Mode: Supports Pairwise Battle for intuitive model ranking.
    • Extensibility: Easy to add custom datasets, models, and evaluation metrics.

  11. Overview of ToolBench-Static benchmark

    main

    ToolBench-Static is a benchmark designed to evaluate an AI model's ability to perform tool use and function calling in a static evaluation setting (no actual tool execution). It tests both tool selection and parameter generation across two subsets:

    • In-domain: Evaluation using tools the model has seen.
    • Out-of-domain: Evaluation using unseen tools.

    Key Metrics:

    • F1 (Primary metric)
    • Act.EM (Action Exact Match)
    • Plan.EM (Plan Exact Match)
    • HalluRate (Measures tool hallucination frequency)
    • Rouge-L

  12. Overview of RAG Evaluation Frameworks

    main

    This survey identifies several key frameworks for evaluating Retrieval-Augmented Generation (RAG) systems:

    • LlamaIndex: Focuses on multimodal RAG evaluation (e.g., assessing both retriever and generator using images and text).
    • LangChain: Provides tutorials for multimodal (image-text) and semi-structured (image-text-table) RAG.
    • Ragas: A no-reference framework for component-level evaluation (retriever vs. generator) using LLMs.
    • RAGChecker: An advanced framework for automatic evaluation, diagnostics, and fine-grained analysis.
    • TruLens: A monitoring and recording tool used to specify feedback and compare application versions via a UI; supports LangChain and LlamaIndex.
    • FlagEmbedding: Focuses on embedding models, re-ranking models, and benchmarking, though it is not a standalone evaluation framework.
    • Continuous Eval: Provides data-driven, modular evaluation for LLM applications, supporting custom metrics and synthetic dataset generation.