LAVIS: Language-Vision Intelligence Library

Img2LLM is a plug-and-play module designed for zero-shot Visual Question Answering (VQA) using frozen Large Language Models (LLMs). It bridges the gap between visual modalities and LLM text processing by converting images into textual prompts. This allows LLMs to perform VQA tasks without requiring end-to-end training.

The workflow consists of four main stages:

1. Image-question matching: Computing relevancy scores for image patches relative to a question to filter out noisy captions.
2. Image captioning: Generating question-guided captions based on the relevancy scores.
3. Question Generation: Creating questions based on synthetic answers and captions.
4. Large Language Model: Using a pre-trained LLM (such as OPT or GPT-3) to perform the final VQA task.

The VQAv2 dataset is designed for the Visual Question Answering (VQA) task. The goal is to provide an accurate natural language answer given an image and an open-ended question about it.

Dataset Characteristics:

• Content: 265,016 images (including COCO and abstract scenes).
• Questions: At least 3 questions per image (average of 5.4).
• Answers: 10 ground truth answers per question and 3 plausible (but likely incorrect) answers.
• Task Type: Free-form and open-ended Visual Question Answering.
• Evaluation: Uses an automatic evaluation metric, with official leaderboards hosted on EvalAI.

Nocaps (Novel Object Captioning at Scale) is a benchmark designed to evaluate how well image captioning models can generalize to visual concepts not present in their training data.

Dataset Details:

• Content: 166,100 human-generated captions describing 15,100 images from Open Images validation and test sets.
• Training Data: Uses COCO image-caption pairs combined with Open Images image-level labels and object bounding boxes.
• The Challenge: Nearly 400 object classes in the test images have little to no associated training captions, forcing models to rely on alternative data sources (like object detection datasets) rather than direct supervision.

Evaluation Metrics: Models are typically evaluated using the following metrics:

• CIDEr
• SPICE

LAVIS (Language-and-VISion Intelligence) is a Python deep learning library designed for multimodal research. It provides a unified interface for over 10 tasks, 20 datasets, and 30+ pretrained models.

Key features include:

• Unified and Modular Interface: Easily leverage or repurpose datasets, models, and preprocessors.
• Off-the-shelf Inference: Access state-of-the-art multimodal understanding and generation capabilities on your own data.
• Reproducible Model Zoo: Replicate and extend SOTA models using provided training recipes.
• Dataset Zoo: Automatic downloading tools for various language-vision datasets.

The AVSD dataset is designed for video-grounded dialogue tasks. It contains over 10,000 dialogues, where each dialogue is grounded in a unique video.

In a typical task, a system must generate responses to user inputs by considering:

1. Dialog History: Previous utterances from both the user and the system.
2. Multimodal Context: Video and audio information that comprise the scene.

In the test split, each sample includes 6 reference dialogue responses for evaluation.

The MSRVTT-QA dataset is a large-scale benchmark designed for Video Question Answering (VideoQA). It is an extension of the MSR-VTT dataset, featuring more complex scenes.

Key Specifications:

• Content: 10K video clips and 243k question-answer pairs.
• Task: Given a video and a natural language question, the model must provide the correct answer.
• Metric: Performance is evaluated using Accuracy (specifically on the test-dev split).

The GQA dataset is a Visual Question Answering (VQA) benchmark.

Task: Given an image and a question, the model must output the correct answer. The dataset specifically tests spatial understanding, multiple reasoning skills, and multi-step inference.

Metrics: While accuracy is the most commonly reported metric, the dataset also evaluates consistency, validity, and plausibility.

The MSRVTT dataset is a large-scale video benchmark designed for video understanding and video-to-text translation tasks. It consists of approximately 10,000 web video clips (totaling 41.2 hours) and 200,000 clip-sentence pairs. Each video clip is annotated with roughly 20 natural language sentences.

Supported Retrieval Tasks

• video-text: Using a video as the query to retrieve corresponding texts from a gallery.
• text-video: Using text as the query to retrieve corresponding videos from a gallery.

The COCO Captioning task involves image captioning: describing the content of an image in words. This is typically implemented using an encoder-decoder framework where an input image is encoded into an intermediate representation and then decoded into a descriptive text sequence.

Evaluation Metrics

Models for this task are typically evaluated using the following metrics:

• BLEU
• CIDEr

The MSVD-QA dataset is designed for the Video Question Answering (VideoQA) task.

Task Definition: Given a video clip and a natural language question, the model must provide the correct answer.

Dataset Composition:

• Video Clips: 1,970
• Question-Answer Pairs: 50,505
• Primary Metric: Accuracy

This dataset is derived from the Microsoft Research Video Description Corpus.