InternVL Documentation

Mixed Preference Optimization (MPO) is a training method designed to enhance the multimodal reasoning and Chain-of-Thought (CoT) performance of Multimodal Large Language Models (MLLMs). It combines three distinct loss components to help the model learn relative preferences, absolute response quality, and the generation process of preferred responses:

1. Preference Loss ($\mathcal{L}_{p}$): Uses DPO (Direct Preference Optimization) to learn the relative preference between chosen and rejected responses.
2. Quality Loss ($\mathcal{L}_{q}$): Uses BCO (Binary Classifier Optimization) to help the model understand the absolute quality of individual responses.
3. Generation Loss ($\mathcal{L}_{g}$): Uses SFT (Supervised Fine-Tuning) loss to help the model learn the generation process of preferred responses.

The total training objective is defined as: $\mathcal{L}=w_{p}\cdot\mathcal{L}{p} + w{q}\cdot\mathcal{L}{q} + w{g}\cdot\mathcal{L}{g}$, where $w{*}$ represents the weights for each component.

The InternVL 2.5 series consists of several multimodal models varying in size, combining an InternViT vision encoder with different LLMs (Qwen 2.5 or InternLM 2.5).

The InternVL evaluation results are obtained using several different codebases depending on the benchmark category. To evaluate InternVL on specific tasks, you must use the corresponding codebase listed below:

Multimodal Reasoning and Mathematics

• MMMU-Pro: Use the local ./mmmu_pro directory.
• AI2D (with mask), ChartQA, DocVQA, InfoVQA: Use the local ./vqa directory.
• CharXiv: Use the CharXiv repository.
• MMMU, MathVista, MATH-Vision, MathVerse, OlympiadBench, AI2D (without mask), OCRBench, SEED-2-Plus, VCR: Use VLMEvalKit.

Multi-Image Understanding

• Mantis Eval: Use the local ./mantis_eval directory.
• MMIU: Use the local ./mmiu directory.
• MIRB: Use the local ./mirb directory.
• BLINK, MuirBench, MMT-Bench: Use VLMEvalKit.

Comprehensive Multimodal Evaluation

• MME: Use the local ./mme directory.
• MMBench: Use the local ./mmbench directory.
• MMVet v2: Use the local ./mmvetv2 directory.
• MMBench v1.1, MMVet, MMStar: Use VLMEvalKit.

Multimodal Hallucination Evaluation

• MMHal-Bench: Use the local ./mmhal directory.
• POPE: Use the local ./pope directory.
• HallBench, CRPE: Use VLMEvalKit.

Visual Grounding

• RefCOCO, RefCOCO+, RefCOCOg: Use the local ./refcoco directory.

Video Understanding

• MVBench: Use the local ./mvbench directory.
• Video-MME, MMBench-Video, MLVU, LongVideoBench: Use VLMEvalKit.
• CG-Bench: Provided by authors.

Other Benchmarks

• Real-World Comprehension (RealWorldQA, MME-RealWorld, WildVision, R-Bench): Use VLMEvalKit.
• Multilingual (MMMB, Multilingual MMBench, MTVQA): Use VLMEvalKit.

InternVL is a suite of multimodal models capable of various visual and cross-modal tasks. Its primary capabilities include:

Visual Perception

• Linear-Probe Image Classification: High-performance image classification using models like InternViT-6B.
• Semantic Segmentation: Precise pixel-level segmentation (e.g., using InternViT-6B with UperNet).
• Zero-Shot Image Classification: Classifying images without specific training on target classes (e.g., ImageNet variants).
• Multilingual Zero-Shot Image Classification: Supporting multiple languages including English (EN), Chinese (ZH), Japanese (JP), Arabic (Ar), and Italian (IT).
• Zero-Shot Video Classification: Classifying video content across datasets like K400, K600, and K700.

Cross-Modal Retrieval

• English Zero-Shot Image-Text Retrieval: Retrieving images from text or vice versa using datasets like Flickr30K and COCO.
• Chinese Zero-Shot Image-Text Retrieval: Specialized retrieval for Chinese language datasets (Flickr30K-CN, COCO-CN).
• Multilingual Zero-Shot Image-Text Retrieval (XTD): Retrieval across multiple languages including English, Spanish, French, Chinese, Italian, Korean, Russian, and Japanese.

Multimodal Dialogue

• Support for interactive dialogue involving both text and visual inputs.