DOCSEARCHbyAlgolia
Submit Docs

lmms-eval

repository·main·Indexed 26 days ago

https://github.com/evolvinglmms-lab/lmms-eval

A unified evaluation toolkit for frontier Large Multimodal Models (LMMs) providing a reproducible pipeline for probing abilities across vision, audio, and video. Version 0.7.2 includes a task catalog of over 100 registered tasks, including AIME, ARC-AGI, and AI2 Reasoning Challenge. It supports a CLI, Web UI, and Python library interface, featuring response caching via SQLite and a framework for implementing custom models and YAML-defined tasks.

Tokens
90K
Snippets
243
Records
468
Agent score
88%

What's inside lmms-eval

  1. Overview of tau2_bench_telecom task

    main

    The tau2_bench_telecom task evaluates agentic loops within the lmms-eval framework.

    Key Technical Details:

    • Output Type: generate_until_agentic
    • Tooling: Uses pure Python functions located in utils.py.
    • Agentic Loop Mechanism: The model interacts by emitting <tool_call>...</tool_call> blocks. The loop continues until the model emits a <submit>...</submit> block, at which point the goal is to have reached the target_state.
    • Integration Note: This serves as a validation of the agentic infrastructure. Full official $\tau^2$-bench domain integration can be achieved by converting official task specifications into lmms-eval docs.

  2. Overview of MMVP (Multimodal Visual Patterns) benchmark

    main
    MMVP is a benchmark designed to identify "CLIP-blind pairs"—images that CLIP perceives as similar despite clear visual differences. It evaluates Vision-Language Models (VLMs) across 9 basic visual patterns, such as orientation, direction, color, and counting. The dataset consists of 300 samples (150 pairs), where each pair contains the same question but with opposite correct answers (A/B).

  3. Overview of PRISMM-Bench tasks

    main

    PRISMM-Bench evaluates LMMs on identifying and resolving inconsistencies in scientific papers. It consists of 7 task variants across 3 categories:

    1. Identification Tasks

    Identify inconsistencies in scientific papers. Output is a single letter (A, B, C, or D).

    • prismm_bench_identification: Uses extracted text parts and corresponding images.
    • prismm_bench_identification_whole_page: Uses full page screenshots (no text context).
    • prismm_bench_identification_whole_doc: Uses up to 5 concatenated/tiled pages (no text context).

    2. Remedy Tasks

    Determine actions to resolve inconsistencies. Output is a single letter (A, B, C, or D).

    • prismm_bench_remedy: Uses extracted text parts and corresponding images.
    • prismm_bench_remedy_whole_page: Uses full page screenshots (no text context).
    • prismm_bench_remedy_whole_doc: Uses up to 5 concatenated/tiled pages (no text context).

    3. Pair-Match Task

    Find which part of a paper creates an inconsistency when combined with a query.

    • prismm_bench_pair_match: Input is a query (text or image) and multiple candidate images. Output is a single letter (A, B, C, or D).

  4. Overview of LSDBench Benchmark

    main
    LSDBench (Long-video Sampling Dilemma Benchmark) is a benchmark for evaluating Vision-Language Models (VLMs) on long-video tasks. It specifically targets the 'Sampling Dilemma': the trade-off between low-density sampling (missing critical information) and high-density sampling (introducing redundancy and slowing inference). The benchmark focuses on videos with dense action sequences that require high Necessary Sampling Density (NSD).

  5. Overview of RealUnify Benchmark

    main

    RealUnify is a benchmark designed to evaluate bidirectional capability synergy in unified multimodal models. It consists of 1,000 human-annotated instances across 10 categories and 32 subtasks. The benchmark evaluates two primary axes:

    1. Understanding Enhances Generation (UEG): Requires reasoning to guide image generation.
    2. Generation Enhances Understanding (GEU): Requires mental simulation to solve reasoning tasks.

    The dataset is hosted on Hugging Face at lmms-lab-eval/RealUnify.

  6. Overview of lmms-eval capabilities

    main

    lmms-eval is a unified evaluation framework for multimodal large language models (LMMs), supporting image, video, and audio modalities. It provides a broad suite of tools including:

    • Benchmark Coverage: 197+ tasks.
    • Model Support: 105+ model implementations.
    • Infrastructure: An HTTP async evaluation server (v0.6) for production-ready job submission and a Web UI (TUI) for interactive configuration.
    • Advanced Features: Statistical confidence intervals (CLT, bootstrap, clustered stderr) and reasoning evaluation using <think>/<answer> structured output with LLM-as-judge.

  7. Overview of the LEMONADE benchmark

    main

    LEMONADE (Language models Evaluation of MOtion aNd Action-Driven Enquiries) is a QA benchmark for evaluating Large Multimodal Models (LMMs) using egocentric video clips from the EPFL-Smart-Kitchen-30 dataset. It contains 36,521 closed-ended QA pairs.

    The benchmark evaluates models across three main groups and six subcategories:

    • Behavior Understanding:
      • Perception: Recognizing perceived actions.
      • Reasoning: Reasoning over unseen behaviors.
    • Long-term Understanding:
      • Summarization: Summarizing over longer clips.
      • Session Properties: Inferring session-level information.
    • Motion & Biomechanics:
      • Physical Attributes: Inferring hand shapes, joint angles, etc.
      • Kinematics: Inferring trajectory velocities.

  8. Overview of AIME tasks

    main
    The AIME (American Invitational Mathematics Examination) tasks evaluate mathematical problem-solving capabilities using competition-level problems. All problems require an integer answer between 0 and 999 inclusive. The implementation supports two main versions: aime_nofigures (problems without diagrams) and aime_figures (problems with diagrams).

  9. Overview of AuxSolidMath benchmark

    main
    AuxSolidMath is a benchmark designed to evaluate solid geometry reasoning capabilities in Large Multimodal Models (LMMs) specifically through the lens of auxiliary line construction. It is based on the paper GeoVLMath: Enhancing Geometry Reasoning via Auxiliary Lines and utilizes the shasha/AuxSolidMath dataset. The benchmark contains 3,018 real-exam solid geometry problems.

  10. ExtremeWhenBench Task Overview

    main

    ExtremeWhenBench is a benchmark for hour-scale natural-language temporal grounding. It consists of 2,273 questions across 194 hour-long videos (averaging 75.7 minutes) sourced from LVBench, MLVU, and VideoMME.

    Metrics:

    • mIoU
    • R@{0.3, 0.5, 0.7}

    Data Format:

    • Predictions are [start, end] intervals in seconds.
    • parse_failures are treated as IoU = 0 to match paper conventions.
    • The dataset includes qid, video_id, source_corpus, question, and correct_interval for evaluation, plus event_summary, category, and youtube_url for analysis.

  11. Understand Throughput Metrics in LMMS-Eval

    main

    LMMS-Eval chat models automatically log detailed timing metrics at the INFO logging level during inference. These metrics provide insights into end-to-end latency, token generation speed, and performance characteristics.

    Core Timing Metrics

    • E2E (End-to-End Latency): Total time from request submission to response completion (seconds).
    • TTFT (Time to First Token): Time from request submission until the first token is generated (seconds).
    • TPOT (Time Per Output Token): Average time to generate each output token after the first (seconds).
    • Speed (Inference Speed): Token generation rate calculated as 1/TPOT (tokens per second).
    • Output Tokens: Number of tokens generated in the response.

    Batch Metrics

    For models processing multiple requests in batches, the following aggregated metrics are provided:

    • Batch Summary: Aggregated metrics across all outputs in a batch.
    • Total Time: Total batch processing time.
    • Total Tokens: Sum of all output tokens in the batch.
    • Avg Speed: Average throughput across the entire batch (tokens/s).

  12. MDPBench task implementation details

    main

    MDPBench evaluates multilingual document-to-Markdown recognition for digital and photographed pages.

    Key Metrics:

    • Text edit distance
    • Formula CDM
    • Table TEDS
    • Overall scores (including language and acquisition type breakdowns)

    Data Source: Uses the Delores-Lin/MDPBench-VLMEvalKit Hugging Face dataset via the train split.

    Model Interaction:

    • Chat-model backends: The task sends a single user message containing the image followed by the benchmark prompt. No system message is inserted.
    • Simple-model backends: Supports legacy doc_to_visual and doc_to_text task interfaces.