DOCSEARCHbyAlgolia
Submit Docs

VLMEvalKit Documentation

repository·main·Indexed 26 days ago

https://github.com/open-compass/vlmevalkit

An open-source toolkit for the one-command evaluation of Large Vision-Language Models (LVLMs) across numerous benchmarks. It supports exact matching, LLM-based answer extraction, and distributed inference via LMDeploy or VLLM. The toolkit includes support for specialized benchmarks such as EgoExoBench, SGI-Bench-1.0, CC-OCR, and MEGA-Bench, and provides a flexible JSON-based configuration system for managing multiple models and datasets.

Tokens
8K
Snippets
25
Records
45
Agent score
88%

What's inside VLMEvalKit

  1. Prepare EgoExoBench video data

    main

    EgoExoBench uses six datasets: Ego-Exo4D, LEMMA, EgoExoLearn, TF2023, EgoMe, and CVMHAT. The evaluation script automatically downloads most datasets, but Ego-Exo4D must be downloaded manually from its official website due to license restrictions.

    To prepare the data manually, create the following directory structure under your data root ([LMUData]):

    [LMUData]/videos/EgoExoBench
├── CVMHAT/
│   └── data/
├── EgoExo4D/
│   └── takes/
├── EgoExoLearn/
├── EgoMe/
├── LEMMA/
├── TF2023/
│   └── data/
├── processed_frames/
└── processed_videos/

  2. Configure VLM settings in vlmeval/config.py

    main

    All VLMs are configured in vlmeval/config.py. Use the model name specified in supported_VLM within that file to select your model. For legacy models like MiniGPT-4 or LLaVA-v1-7B, you may need to manually configure the code or model weight root in the config file.

    To verify a model configuration before running a full evaluation, use the vlmutil check command: vlmutil check {MODEL_NAME}

  3. Obtain MEGA-Bench breakdown analysis

    main

    To generate a multi-dimensional breakdown analysis (consistent with the MEGA-Bench leaderboard format), follow these steps:

    1. Ensure both megabench_score_core.json and megabench_score_open.json are moved into the same directory.
    2. Run the derive_breakdown_results.py script located in vlmeval/dataset/utils/megabench/tools pointing to that directory.

    The analysis results will be saved in an analysis subdirectory within your input directory.

    # Run the metrics for the open-ended set
cd vlmeval/dataset/utils/megabench/tools
python3 derive_breakdown_results.py  --input_dir your/dir/to/megabench_scores

  4. Use the Config System for flexible evaluation

    main

    While run.py supports simple evaluation using --model and --data arguments, you can use a JSON configuration file for more complex scenarios involving multiple models or datasets with different settings. Use the --config argument to pass the path to your JSON file.

    Config JSON Structure

    The JSON file must contain two top-level dictionary keys: model and data.

    1. model Configuration

    Each entry in the model dictionary uses a user-defined name as the key. The value is a dictionary containing:

    • class: The class name of the model (defined in vlmeval/vlm/__init__.py for open-source models or vlmeval/api/__init__.py for API models).
    • Other kwargs: Model-specific parameters (e.g., temperature, img_detail). Note that the model argument is required by most model classes.
    • Shortcut: If a model is already defined in supported_VLM within vlmeval/config.py, you can use an empty dictionary {} to use its default settings.

    2. data Configuration

    Each entry in the data dictionary uses a name (ideally the official dataset name) as the key. The value is a dictionary containing:

    • class: The class name of the dataset (defined in vlmeval/dataset/__init__.py).
    • Other kwargs: Dataset-specific parameters (e.g., nframe, fps, use_subtitle). The dataset argument is typically required. For video datasets, nframe or fps is usually required.
    • Shortcut: If a dataset is defined in supported_video_datasets within vlmeval/dataset/video_dataset_config.py, you can use an empty dictionary {} to use its default settings.

    Execution and Output

    Run the evaluation using:

    python run.py --config config.json

    Results are saved in $WORK_DIR using the pattern: {$WORK_DIR}/{$MODEL_NAME}/{$MODEL_NAME}_{$DATASET_NAME}_*.

    {
    "model": {
        "GPT4o_20240806_T00_HIGH": {
            "class": "GPT4V",
            "model": "gpt-4o-2024-08-06",
            "temperature": 0,
            "img_detail": "high"
        },
        "GPT4o_20241120": {}
    },
    "data": {
        "MME-RealWorld-Lite": {
            "class": "MMERealWorld",
            "dataset": "MME-RealWorld-Lite"
        },
        "Video-MME_16frame_subs": {
            "class": "VideoMME",
            "dataset": "Video-MME",
            "nframe": 16,
            "use_subtitle": true
        }
    }
}

  5. Prepare code for contribution using pre-commit

    main

    Before submitting a Pull Request to VLMEvalKit, run the pre-commit checks to ensure code quality. Follow these steps within the repository directory:

    1. Install pre-commit via pip.
    2. Install the git hooks.
    3. Run the checks on all files.
    # Under the directory of VLMEvalKit, install the pre-commit hook:
pip install pre-commit
pre-commit install
pre-commit run --all-files
# Then you can commit your code.

  7. Evaluate models on EgoExoBench

    main

    Use run.py to evaluate Vision-Language Models (VLMs) on the EgoExoBench multiple-choice questions (MCQs). Use torchrun for lightweight models and standard python for larger models requiring more memory.

    # For lightweight vision-language models
torchrun --nproc-per-node=1 run.py \
    --data EgoExoBench_MCQ \
    --model Qwen2.5-VL-7B-Instruct-ForVideo

# For larger models with higher memory usage
python run.py \
    --data EgoExoBench_MCQ \
    --model Qwen2.5-VL-72B-Instruct-ForVideo

  8. Run CC-OCR batch inference and evaluation

    main

    To perform batch inference and evaluation for the CC-OCR benchmark, execute the run.py script followed by the ccocr_evaluator/common.py script. The benchmark is divided into four main categories: multi_scene_ocr, multi_lan_ocr, doc_parsing, and kie.

    For each category, you must specify the --data datasets, the --model name, and a --work-dir directory. After running the inference via run.py, run the evaluator script to generate the final summary.

    Note: Ensure you are in the root directory of VLMEvalKit when executing these commands.

    # Example for KIE (Key Information Extraction) subset
MODEL_NAME="QwenVLMax"
OUTPUT_DIR="/your/path/to/output_dir"
SUB_OUTPUT_DIR=${OUTPUT_DIR}/kie

# 1. Run inference and evaluation tasks in batch
python run.py --data CCOCR_Kie_Sroie2019Word CCOCR_Kie_Cord CCOCR_Kie_EphoieScut CCOCR_Kie_Poie CCOCR_Kie_ColdSibr CCOCR_Kie_ColdCell --model ${MODEL_NAME} --work-dir ${SUB_OUTPUT_DIR} --verbose

# 2. Run the CC-OCR evaluator to generate summary
python vlmeval/dataset/utils/ccocr_evaluator/common.py ${SUB_OUTPUT_DIR}

  9. Get started with VLMEvalKit

    main

    To begin using VLMEvalKit, follow the recommended path based on your goals:

    1. New Users: Follow the Quickstart guide to set up your environment and run a mini-experiment. This is the best way to familiarize yourself with the basic evaluation process.
    2. Advanced Users: If you need to extend the toolkit (e.g., adding new datasets or models), refer to the Advanced Tutorial, which includes documentation on the Development Guide and the Config System.

  10. Configure API keys via .env file

    main

    To use proprietary VLMs (e.g., GPT-4v, Gemini) or use LLMs as judges/choice extractors, set up your API keys. You can either set them as environment variables or create a .env file in the $VLMEvalKit root directory.

    Note: If no LLM judge key is provided, VLMEvalKit defaults to exact matching mode (searching for 'Yes', 'No', 'A', 'B', etc.), which is only applicable to Yes-or-No and Multi-choice tasks.

    # Example .env file content
DASHSCOPE_API_KEY=
GOOGLE_API_KEY=
OPENAI_API_KEY=
OPENAI_API_BASE=
STEPAI_API_KEY=
REKA_API_KEY=
GLMV_API_KEY=
CW_API_BASE=
CW_API_KEY=
SENSENOVA_API_KEY=
HUNYUAN_SECRET_KEY=
HUNYUAN_SECRET_ID=
LMDEPLOY_API_BASE=
MINIMAX_API_KEY=
EVAL_PROXY=

  11. Start an LMDeploy inference service

    main

    Start an LMDeploy API server to host your model.

    Important Requirements:

    • You must specify the --model-name flag. This allows VLMEvalKit to select the correct prompt construction strategy (e.g., for InternVL2's handling of HallusionBench) when communicating via the API.
    • If you specify a custom --server-port, you must set the LMDEPLOY_API_BASE environment variable to point to your server's address.
    lmdeploy serve api_server OpenGVLab/InternVL2-8B --model-name InternVL2-8B